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Highlights 


Highlights  of  GAO-13-103,  a  report  to 
Committee  on  Armed  Services,  U.S.  Senate 


WEAPONS  ACQUISITION  REFORM 

Reform  Act  Is  Helping  DOD  Acquisition  Programs 
Reduce  Risk,  but  Implementation  Challenges  Remain 


Why  GAO  Did  This  Study 

For  the  past  3  years,  DOD  has  been 
implementing  the  Reform  Act 
requirements  which  are  aimed  at 
helping  weapon  acquisition  programs 
establish  a  solid  foundation  from  the 
start.  This  helps  to  prevent  cost 
growth,  thus  helping  the  Defense  dollar 
go  further.  This  is  the  third  in  a  series 
of  GAO  reports  on  the  Reform  Act. 

GAO  was  asked  to  determine 
(1)  DOD’s  progress  in  implementing 
Reform  Act  provisions;  (2)  the  impact 
the  Reform  Act  has  had  on  specific 
acquisition  programs;  and 
(3)  challenges  remaining.  To  do  this, 
GAO  analyzed  documents  and 
interviewed  officials  from  the  four  OSD 
offices  created  as  a  result  of  the 
Reform  Act,  other  DOD  offices,  the 
military  services,  and  1 1  weapon 
acquisition  programs  we  chose  as 
case  studies.  Case  study  programs 
were  selected  based  on  their 
development  status  and  interaction 
with  the  four  OSD  offices.  Results 
cannot  be  generalized  to  all  DOD 
weapon  acquisition  programs. 

What  GAO  Recommends 

GAO  recommends  DOD  develop 
additional  cost  estimating  and  Reform 
Act  implementation  guidance;  make 
lessons  learned  available  to  the 
acquisition  community;  and  assess  the 
adequacy  of  the  military  services’ 
systems  engineering  and 
developmental  testing  workforce.  DOD 
generally  concurred  with  the 
recommendations.  GAO  clarified  one 
recommendation  to  make  it  clear  that 
DOD  needs  to  designate  an  office(s) 
within  the  Acquisition,  Technology  and 
Logistics  organization  to  provide 
practical  Reform  Act  implementation 
guidance  to  program  offices. 

View  GAO-13-103.  For  more  information, 
contact  Michael  J.  Sullivan  at  (202)  512-4841 
or  sullivanm@gao.gov. 


What  GAO  Found 

The  Department  of  Defense  (DOD)  has  taken  steps  to  implement  fundamental 
Weapon  Systems  Acquisition  Reform  Act  of  2009  (Reform  Act)  provisions, 
including  those  for  approving  acquisition  strategies  and  better  monitoring 
weapon  acquisition  programs.  DOD  is  also  continuing  to  take  additional  steps  to 
strengthen  policies  and  capabilities.  Some  provisions,  such  as  issuing  guidance 
for  estimating  operating  and  support  costs,  are  being  implemented. 

GAO’s  analysis  of  11  weapon  acquisition  programs  showed  the  Reform  Act  has 
reinforced  early  attention  to  requirements,  cost  and  schedule  estimates,  testing, 
and  reliability.  For  example,  prior  to  starting  development,  an  independent  review 
team  raised  concerns  about  the  Ground  Combat  Vehicle  program’s  many 
requirements  and  the  risks  associated  with  its  7-year  schedule.  Subsequently, 
the  Army  reduced  the  number  of  requirements  by  about  25  percent  and 
prioritized  them,  giving  contractors  more  flexibility  in  designing  solutions.  In 
addition,  the  developmental  test  and  evaluation  office — resulting  from  the  Reform 
Act — used  test  results  to  help  the  Joint  Light  Tactical  Vehicle  program  develop  a 
more  realistic  reliability  goal  and  a  better  approach  to  reach  it.  Shown  below  are 
areas  where  the  Reform  Act  influenced  several  programs  in  GAO’s  review. 


Reform  Act  Influence  on  Select  Case  Study  Programs 

Program 

Requirements  Cost  and  schedule 

Testing 

Reliability 

Ohio  Class  Replacement 

✓ 

Ground  Combat  Vehicle 

s 

Joint  Light  Tactical  Vehicle 

s 

V 

Ship  to  Shore  Connector 

s 

KC-46  Tanker 

s 

Source:  GAO  analysis  of  DOD  data. 


While  DOD  has  taken  steps  to  implement  most  of  the  fundamental  Reform  Act 
provisions,  some  key  efforts  to  date  have  been  primarily  focused  on  DOD's 
largest  weapon  acquisition  programs.  DOD  faces  five  challenges — organizational 
capability  constraints,  the  need  for  additional  guidance  on  cost  estimating  and 
Reform  Act  implementation,  the  uncertainty  about  the  sufficiency  of  systems 
engineering  and  developmental  testing  resources,  limited  dissemination  of 
lessons  learned,  and  cultural  barriers  between  the  Office  of  the  Secretary  of 
Defense  (OSD)  and  the  military  services — that  limit  its  ability  to  broaden  the 
Reform  Act’s  influence  to  more  programs.  Service  officials  believe  additional 
guidance  is  needed  to  improve  their  cost  estimates  and  other  implementation 
efforts.  They  also  believe  that  lessons  learned  from  programs  that  experience 
significant  cost  and  schedule  increases  should  be  shared  more  broadly  within  the 
acquisition  community.  These  challenges  seem  straightforward  to  address,  but 
they  may  require  more  resources,  which  have  been  difficult  to  obtain.  Ensuring 
the  services  have  key  leaders  and  staff  dedicated  to  systems  engineering  and 
developmental  testing  activities,  such  as  chief  engineers  at  the  service  level  and 
technical  leads  on  programs,  as  well  as  breaking  down  cultural  barriers  are  more 
difficult  to  address.  They  will  require  continued  monitoring  and  attention  by  the 
Under  Secretary  for  Acquisition,  Technology  and  Logistics,  service  acquisition 
executives,  and  offices  established  as  a  result  of  the  Reform  Act  to  address. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


December  14,  2012 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

For  several  decades,  GAO  has  reported  on  poor  outcomes 
encompassing  cost  and  schedule  growth  on  the  Department  of  Defense’s 
(DOD)  major  weapon  acquisition  programs.  Many  problems  can  be  traced 
to  a  culture  where  the  military  services  begin  programs  with  inflexible 
requirements,  immature  technologies,  and  overly  optimistic  cost  and 
schedule  estimates.  Given  pressures  to  reduce  spending  across  the 
government,  including  DOD,  finding  ways  to  prevent  or  mitigate  cost 
growth  is  crucial  to  U.S.  national  security.  A  solid  program  foundation 
using  good  developmental  testing  and  systems  engineering,  and  reliable 
cost  estimates  is  needed  in  order  to  help  avoid  cost  overruns  and 
promote  better  acquisition  outcomes.  There  have  been  numerous 
attempts  in  the  past  to  improve  DOD  acquisition  outcomes,  including  the 
Packard  Commission  and  the  Goldwater-Nichols  Act  in  the  1980s  and 
Federal  Acquisition  Streamlining  Act  of  1994.1  More  recently,  Congress 
passed  the  Weapon  Systems  Acquisition  Reform  Act  of  2009  (Reform 
Act)2  to  improve  the  way  weapon  systems  are  acquired  and  avoid  cost 
and  schedule  overruns. 

In  2009,  the  Senate  Armed  Services  Committee  asked  us  to  begin 
reporting  on  DOD’s  implementation  of  Reform  Act  provisions  and  the 
impact  the  Reform  Act  has  had  on  weapon  acquisition  programs.  This  is 
our  third  report  addressing  these  topics.  The  first  report  focused  on 
DOD’s  initial  efforts  to  implement  Reform  Act  provisions  for  systems 
engineering  and  developmental  testing,  including  the  placement  of  new 


1  Final  Report  to  the  President  by  the  President’s  Blue  Ribbon  Commission  on  Defense 
Management  (June  1986);  Goldwater-Nichols  Department  of  Defense  Reorganization  Act 
of  1986,  Pub.  L.  No.  99-433;  and  Pub.  L.  No.  103-355  (FASA). 

2  Pub.  L.  No.  111-23,  as  amended  by  the  Ike  Skelton  National  Defense  Authorization  Act 
for  Fiscal  Year  2011,  Pub.  L.  No.  111-383  §§  813  and  1075,  and  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2012,  Pub.  L.  No.  112-81  §§819  and  837. 
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offices  for  these  activities  within  the  Office  of  the  Secretary  of  Defense 

(OSD).3  Our  second  report  examined  the  challenges  the  services  face  as 

they  try  to  strengthen  systems  engineering  and  developmental  testing 

activities  on  weapon  acquisition  programs.4  This  report  examines 

(1)  DOD’s  progress  in  implementing  Reform  Act  provisions;  (2)  the  impact 

the  Reform  Act  has  had  on  specific  acquisition  programs;  and 

(3)  challenges  remaining  in  improving  the  weapons  acquisition  process. 

To  determine  DOD’s  progress  in  implementing  Reform  Act  provisions  we 
interviewed  officials  and  analyzed  documentation  from  the  four  offices 
whose  leadership  was  established  within  OSD  as  a  result  of  the  Reform 
Act — Systems  Engineering  (SE),  Developmental  Test  and  Evaluation 
(DT&E),  Cost  Assessment  and  Program  Evaluation  (CAPE),  and 
Performance  Assessments  and  Root  Cause  Analyses  (PARCA).5  We 
focused  our  review  on  the  offices’  implementation  of  four  fundamental 
Reform  Act  provisions:  developing  policy  and  guidance;  approving 
acquisition  documents;  monitoring  programs  and  conducting  program 
assessments;  and  developing  performance  measures.  We  also  analyzed 
documentation  we  collected  on  selected  weapon  acquisitions.  To 
determine  the  Reform  Act’s  impact  on  specific  defense  acquisition 
programs,  we  chose  as  case  studies,  1 1  weapon  systems  at  various 
points  in  the  development  process  based  on  recommendations  by  OSD 
officials  and  GAO’s  previous  evaluations  of  these  programs.  At  the  time  of 
our  case  study  selection,  all  of  the  weapon  systems  in  our  review  had 
been  given  Milestone  A  approval.  This  approval  allows  the  programs  to 
start  the  technology  development  phase  of  the  acquisition  process.  Four 
of  the  1 1  weapon  systems  had  not  yet  received  Milestone  B  approval. 

This  approval  signifies  the  start  of  engineering  and  manufacturing 
development  activities.  We  believe  these  four  programs  offer  the  best 
insight  into  how  the  OSD  offices  and  Reform  Act  policies  are  influencing 


3  GAO,  Defense  Acquisitions:  DOD  Needs  to  Develop  Performance  Criteria  to  Gauge 
Impact  of  Reform  Act  Changes  and  Address  Workforce  Issues,  GAO-10-774  (Washington, 
D.C.:  July  29,  2010). 

4  GAO,  Weapons  Acquisition  Reform:  Actions  Needed  to  Address  Systems  Engineering 
and  Developmental  Testing  Challenges,  GAO-1 1-806  (Washington,  D.C.:  Sept.  19,  2011). 

5  The  Director  of  Systems  Engineering  and  the  Director  of  Developmental  Test  and 
Evaluation  offices  were  originally  established  as  a  result  of  the  Reform  Act.  These  offices 
have  since  been  renamed  as  the  offices  of  the  Deputy  Assistant  Secretary  of  Defense  for 
Systems  Engineering  and  the  Deputy  Assistant  Secretary  of  Defense  for  Developmental 
Test  and  Evaluation. 
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acquisition  strategies.  The  other  seven  programs  were  conducting 
engineering  and  manufacturing  development  activities  or  had  been  given 
Milestone  C  approval  to  begin  production  and  deployment  phase 
activities.  Each  of  these  programs  provided  different  insights  in  their 
interactions  with  the  OSD  offices,  but  some  Reform  Act  provisions,  such 
as  conducting  preliminary  design  reviews  prior  to  Milestone  B,  may  not 
apply  to  these  programs  since  they  were  beyond  development  start.  A 
complete  list  of  programs  we  reviewed  is  provided  below. 


Table  1:  Programs  Selected  for  Case  Study  Review 

Programs  that  had  not  started  development  at  the  time  of  our  case  study  selection 

System  development  start  date 

Ground  Combat  Vehicle 

December  201 3  (estimated) 

Joint  Light  Tactical  Vehicle3 

August  2012 

Ohio  Class  Replacement  submarine 

August  2016  (estimated) 

Ship  to  Shore  Connector3 

July  2012 

Programs  that  were  in  development  at  the  time  of  our  case  study  selection 

Global  Hawk  unmanned  aerial  vehicle 

March  2001 

Gray  Eagle  unmanned  aerial  vehicle 

April  2005 

Joint  Strike  Fighter 

October  2001 

KC-46  Tanker  aircraft 

February  201 1 

Littoral  Combat  Ship  Seaframe 

February  201 1 

Remote  Minehunting  System 

December  1999 

Small  Diameter  Bomb  Increment  II 

July  2010 

Source:  GAO  analysis. 

aDuring  the  course  of  our  review,  the  Joint  Light  Tactical  Vehicle  and  Ship  to  Shore  Connector 
programs  held  a  Milestone  B  review  and  received  approval  to  start  development. 


To  determine  challenges  remaining  for  implementing  the  Reform  Act,  we 
conducted  our  own  assessment  and  interviewed  the  leaders  of  the  OSD 
offices  established  as  a  result  of  the  Reform  Act,  military  service 
acquisition  chiefs,  and  program  managers.  These  officials  are  responsible 
for  advising  and  overseeing  weapon  acquisition  program  development, 
funding  and  developing  new  weapon  acquisition  programs,  and  executing 
the  day-to-day  development  plans,  respectively.  We  also  solicited  the 
opinions  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology 
and  Logistics  and  other  senior  OSD  leaders.  Additional  information  about 
our  scope  and  methodology  can  be  found  in  appendix  I. 

We  conducted  this  performance  audit  from  January  2012  to  December 
2012  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
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obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 


Background 


In  May  2009,  Congress  passed  the  Weapon  Systems  Acquisition  Reform 
Act  of  2009  (Reform  Act)  in  an  effort  to  improve  the  way  weapon  systems 
are  acquired  and  avoid  further  cost  overruns  on  such  programs.  When 
signing  the  Reform  Act  into  law,  the  President  stated  that  its  purpose  was 
to  limit  weapon  system  cost  overruns  and  that  it  would  strengthen 
oversight  and  accountability  by  appointing  officials  who  will  closely 
monitor  the  weapons  systems  acquisition  process  to  ensure  that  costs 
are  controlled. 

Four  offices  were  established  as  a  result  of  the  Reform  Act:  SE,  DT&E, 
CAPE  and  PARCA.  The  SE  and  CAPE  offices  existed  under  other 
organizational  titles  prior  to  the  Reform  Act.  Staffing  levels,  following  the 
Reform  Act,  remained  relatively  stable  for  both  of  these  offices,  but  some 
reorganization  was  necessary  to  reflect  new  Reform  Act  responsibilities. 
The  DT&E  and  PARCA  offices  were  newly  established.  The  key  roles  and 
responsibilities  of  these  four  offices  as  outlined  in  the  Reform  Act  are 
explained  below: 
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Table  2:  Key  Responsibilities  of  Offices  Established  as  a  Result  of  the  Reform  Act 


Office 

Primary  responsibilities 

Systems  Engineering 

serves  as  principal  advisor  to  the  Secretary  of  Defense  and  the  Under  Secretary  of  Defense 

for  Acquisition,  Technology  and  Logistics  on  systems  engineering  activities  in  the  department 

develops  systems  engineering  and  development  planning  guidance  for  DOD 

reviews  and  approves  major  defense  acquisition  program  systems  engineering  plans 

monitors  major  defense  acquisition  program  systems  engineering  activities 

reports  to  Congress  annually  on  systems  engineering  organization,  capabilities,  and  activities 

Developmental  Test  and 
Evaluation 

serves  as  principal  advisor  to  the  Secretary  of  Defense  and  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  on  developmental  test  and  evaluation  activities 

develops  developmental  test  and  evaluation  guidance  for  DOD 

reviews  and  approves  major  defense  acquisition  program  developmental  test  and  evaluation 
plans 

monitors  and  reviews  acquisition  program  developmental  test  and  evaluation  activities  of 
major  defense  acquisition  programs 

reports  to  Congress  annually  on  developmental  test  and  evaluation  organization,  capabilities, 
and  activities 

Cost  Assessment  and  Program 
Evaluation 

serves  as  principal  advisor  to  the  Secretary  of  Defense  and  other  senior  officials  on  matters 
related  to  cost  analysis  and  the  planning  and  programming  phases  of  the  planning, 
programming,  budgeting,  and  execution  system 

develops  independent  cost  estimates  for  major  defense  acquisition  programs  prior  to  major 
milestone  decisions  and  updates  independent  cost  estimates  when  programs  have  exceeded 
critical  cost  thresholds,  known  as  Nunn  McCurdy  breaches 

reviews  existing  systems  and  methods  for  tracking  and  assessing  operation  and  support 
costs  on  major  defense  acquisition  programs 

develops  analysis  of  alternative  study  guidance  for  major  defense  acquisition  programs 
approves  the  analysis  of  alternatives  study  plan  for  each  major  defense  acquisition  program 

Performance  Assessments  and 
Root  Cause  Analyses 

assesses  major  acquisition  program  performance  through  independent  analyses  and  through 
the  Defense  Acquisition  Executive  Summary  process 

identifies  the  root  causes  of  cost  growth  and  other  problems  on  programs  that  experience  a 
critical  Nunn  McCurdy  cost  breach 

Source:  GAO  Analysis  of  Weapon  Systems  Acquisition  Reform  Act  of  2009. 

Each  of  the  offices  has  varying  levels  of  interaction  with  defense 
acquisition  programs.  For  example,  the  SE  and  DT&E  offices  have 
ongoing  interaction  with  acquisition  programs  throughout  development  in 
the  form  of  program  reviews,  working  group  activities,  and  other  program 
meetings.  They  also  coordinate  program  documents  in  preparation  for 
major  milestone  reviews.  CAPE  issues  guidance  to  programs  on  how  to 
conduct  an  analysis  of  alternatives  at  the  beginning  of  the  acquisition 
process.  The  office  approves  the  analysis  of  alternative  study  plan  that  is 
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developed  based  on  its  guidance.  It  then  develops  independent  cost 
estimates  for  major  milestone  reviews  and  in  the  event  that  an  acquisition 
program  experiences  a  Nunn-McCurdy  breach.6  According  to  PARCA,  it 
assesses  all  major  defense  acquisition  programs  at  least  once  per  quarter 
or  when  requested  by  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics,  and  disseminates  this  information  to  senior 
leaders.  The  office  also  interacts  with  specific  programs  if  they  experience 
a  Nunn-McCurdy  breach.  In  these  cases,  the  office  assesses  program 
performance  not  less  than  semi-annually  until  1  year  after  it  receives  a 
new  milestone  approval. 

In  addition  to  the  new  organizational  requirements,  the  Reform  Act 
requires  DOD  to  ensure  that  the  acquisition  strategy  for  major  defense 
acquisition  programs  includes  measures  to  ensure  competition  or  the 
option  of  competition  throughout  the  program  life  cycle.  This  could  include 
strategies  such  as  maintaining  two  sources  for  a  system  (dual-sourcing) 
and  breaking  requirements  for  supplies  or  services  previously  provided  or 
performed  under  a  single  contract  into  separate  smaller  contracts 
(unbundling  of  contracts).7  Major  defense  acquisition  programs  are  also 
required  to  provide  for  competitive  prototyping — where  two  or  more 
competing  teams  produce  prototypes  before  a  design  is  selected  for 
further  development — prior  to  Milestone  B  unless  a  waiver  is  properly 


6  10  U.S.C.  §  2433  establishes  the  requirement  for  the  DOD  to  prepare  unit  cost  reports 
on  major  defense  acquisition  programs  or  designated  subprograms.  If  a  program  exceeds 
cost  growth  thresholds  specified  in  the  law,  this  is  commonly  referred  to  as  a  Nunn- 
McCurdy  breach,  which  DOD  is  required  to  report  to  Congress  and,  if  applicable,  submit  a 
certification  to  Congress  in  order  to  continue  the  program,  in  accordance  with  10  U.S.C. 

§  2433a. 

7  Pub.  L.  No.  111-23  §202. 


Page  6 


GAO-13-103  Weapons  Acquisition  Reform 


granted  by  the  milestone  decision  authority,8  and  to  meet  the  following 

Milestone  B  certification  requirements,  including:9 

•  Appropriate  trade-offs  among  cost,  schedule,  and  performance 
objectives  have  been  made  to  ensure  the  program  is  affordable; 

•  A  preliminary  design  review  and  formal  post-preliminary  design  review 
assessment  have  been  conducted  and  on  the  basis  of  such 
assessment  the  program  demonstrates  a  high  likelihood  of 
accomplishing  its  intended  mission; 

•  Technology  has  been  demonstrated  in  a  relevant  environment  on  the 
basis  of  an  independent  review  and  assessment  by  the  Assistant 
Secretary  of  Defense  for  Research  and  Engineering; 

•  Reasonable  cost  and  schedule  estimates  have  been  developed  to 
execute,  with  concurrence  of  the  Director  of  CAPE,  the  program’s 
product  development  and  production  plan; 

•  Funding  is  available  to  execute  the  program’s  product  development 
and  production  plan; 

•  DOD  has  completed  an  analysis  of  alternatives  with  respect  to  the 
program;  and 

•  The  Joint  Requirements  Oversight  Council10  has  approved  program 
requirements,  including  an  analysis  of  the  operational  requirements. 


8  Pub.  L.  No.  111-23  §  203(a).  Specifically,  the  Reform  Act  required  DOD  to  modify  its 
guidance  relating  to  the  operation  of  its  acquisition  system  to  incorporate  these 
competitive  prototyping  provisions.  DOD  did  so  through  Directive-Type  Memorandum 
(DTM)  09-027,  “Implementation  of  Weapon  System  Acquisition  Reform  Act  of  2009  (Dec. 
4,  2009,  incorporating  Change  3,  Dec.  9,  2011).  Major  defense  acquisition  programs  are 
those  estimated  by  DOD  to  require  an  eventual  total  expenditure  for  research, 
development,  test,  and  evaluation,  including  all  planned  increments,  of  more  than  $365 
million,  or  for  procurement,  including  all  planned  increments,  of  more  than  $2.19  billion  in 
fiscal  year  2000  constant  dollars  or  designated  by  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics.  The  Milestone  Decision  Authority  for  major 
defense  acquisition  programs  is  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics,  the  head  of  a  DOD  component,  or  if  delegated  the  component 
acquisition  executive. 

9  Pub.  L.  No.  111-23;  various  sections,  codified  at  10  U.S.C.  §  2366b.  The  Reform  Act 
revised  the  Milestone  B  certification  requirements  for  trade-offs,  preliminary  design, 
technology  demonstration,  and  reasonable  cost  and  schedule  estimates.  The  remaining 
Milestone  B  certification  requirements  bulleted  here  were  unrevised  by  the  Reform  Act. 

10  The  Joint  Requirements  Oversight  Council  is  an  advisory  council  to  the  Chairman  of  the 
Joint  Chiefs  of  Staff  with  the  responsibility  to:  (1 )  identify,  assess,  and  approve  joint 
military  requirements;  (2)  assist  acquisition  officials  in  identifying  alternatives  to  acquisition 
programs;  and  (3)  assist  the  Chairman  of  the  Joint  Chiefs  of  Staff  in  assigning  priority  for 
joint  military  requirements. 
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The  Reform  Act  also  requires  the  Joint  Requirements  Oversight  Council 
to  ensure  trade-offs  among  cost,  schedule,  and  performance  objectives 
are  considered  for  joint  military  requirements.11  GAO  previously  reported 
that  the  Council  considered  trade-offs  made  by  the  military  services 
before  validating  requirements,  but  the  military  services  did  not 
consistently  provide  high-quality  resource  estimates  to  the  Council  for 
proposed  programs  in  fiscal  year  2010.  We  also  found  that  the  Council 
did  not  prioritize  requirements,  consider  redundancies  across  proposed 
programs,  or  prioritize  and  analyze  capability  gaps  in  a  consistent 
manner.12 

DOD  Has 

Implemented  Most  of 
the  Fundamental 
Reform  Act 

Provisions  and  Is 
Strengthening 
Acquisition  Activities 

DOD  has  implemented  most  of  the  fundamental  Reform  Act  provisions  as 
required  and  is  taking  additional  steps  to  strengthen  acquisition  reviews, 
policies,  and  capabilities.  Offices  established  as  a  result  of  the  Reform 

Act  are  continuing  to  issue  policies,  review  and  approve  relevant 
acquisition  documents,  monitor  weapon  acquisition  program  activities, 
and  develop  performance  measures.  In  addition,  all  four  of  the  major 
defense  acquisition  programs  we  reviewed  that  had  not  started 
development  when  we  selected  our  case  studies  plan  to  implement 

Reform  Act  provisions  regarding  preliminary  design  reviews,  competitive 
prototyping,  and  competition.  Also,  some  provisions,  such  as  issuance  of 
guidance  on  estimating  operating  and  support  costs,  by  the  CAPE,  are 
still  in  the  process  of  being  completed.  Finally,  we  found  that  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  has 
revised  the  defense  acquisition  review  process  to  consider  additional 
knowledge  collected  on  programs  earlier  and  efforts  are  being  made  to 
strengthen  acquisition  policies  and  capabilities. 

DOD  Is  Continuing  to 
Implement  Reform  Act 
Provisions 

The  offices  established  as  a  result  of  the  Reform  Act — SE,  DT&E,  CAPE, 
and  PARCA — are  continuing  to  make  progress  in  implementing  four 
fundamental  Reform  Act  provisions  aimed  at  strengthening  acquisition 
outcomes  and  oversight  of  weapon  acquisition  programs.  Specifically,  the 
offices  are  (1)  developing  policy  and  guidance  to  the  military  services  for 
conducting  work  in  their  respective  areas,  (2)  approving  acquisition 

11  Pub.  L.  No.  111-23  §201. 

12  GAO,  DOD  Weapon  Systems:  Missed  Trade-off  Opportunities  During  Requirements 
Reviews,  GAO-1 1-502,  (Washington,  D.C.:  June  16,  2011). 
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documents  prior  to  milestone  reviews,  (3)  monitoring  and  assessing 
weapon  acquisition  program  activities  on  a  consistent  basis,  and 
(4)  developing  performance  measures  to  assess  acquisition  program 
activities.  Figure  1  provides  the  status  of  DOD  efforts  to  implement  the 
four  fundamental  provisions.  Some  offices  are  still  in  the  process  of 
completing  a  few  of  these  provisions.  For  example,  CAPE  and  PARCA 
are  in  the  process  of  developing  policies  and  guidance  for  their  respective 
areas  and  DT&E  is  in  the  process  of  establishing  performance  measures 
that  can  be  used  to  assess  weapon  acquisition  program  activities.  The 
office  piloted  the  performance  measures  on  two  major  defense  acquisition 
programs  and  reported  that  they  are  currently  applying  them  to  over  40 
programs.  Note  that  some  activities  related  to  approving  documents  and 
monitoring  or  assessing  programs  require  on-going  efforts  on  the  part  of 
some  of  the  offices. 
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Figure  1:  DOD’s  Progress  in  Implementing  Four  Fundamental  Reform  Act  Provisions 


Interactive  Graphic 


Click  on  status  to  show  Reform  Act  provisions.  Click  on  the  “®"  to  clear.  See  appendix  II  for  the  non-interactive,  printer-friendly  version. 
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Source:  GAO  analysis  of  DOD  data. 

3  The  Reform  Act  does  not  specify  a  date  of  completion  for  developing  policy  and  guidance. 


bDOD  Directive-Type  Memorandum  (DTM)  11-003,  “Reliability  Analysis,  Planning,  Tracking, 
and  Reporting,”  (Mar.  21,  2011);  DOD  DTM  10-017,  “Development  Planning  to  Inform  Materiel 
Development  Decision  Reviews  and  Support  of  Analysis  of  Alternatives,”  (Sept.  13,  2010, 
Incorporating  Change  2,  Dec.  9,  2011);  DOD  Instruction  5134.16,  “Deputy  Assistant  Secretary 
of  Defense  for  Systems  Engineering  (DASD(SE)),”  (Aug.  19,  2011). 
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We  also  found  evidence  that  major  defense  acquisition  programs  are 
integrating  Reform  Act  provisions  in  their  acquisition  strategies.  The  four 
weapon  acquisition  programs  we  reviewed  that  had  not  started 
development  activities  when  we  began  our  review  plan  to  implement 
Reform  Act  provisions  related  to  preliminary  design  reviews,  competitive 
prototyping,  and  competition.  For  example,  the  Ground  Combat  Vehicle 
has  two  contractors  developing  competitive  prototypes  of  two  key 
subsystems  to  support  technology  development.  The  program  intends  to 
conduct  preliminary  design  reviews  on  both  contractors’  designs  prior  to 
Milestone  B  and  to  conduct  full  and  open  competition  through  Milestone 
C.  Similarly,  according  to  program  officials,  the  Joint  Light  Tactical 
Vehicle  program  had  three  contractors  develop  full-system  prototypes 
during  the  technology  development  phase  and  held  preliminary  design 
reviews  on  each  contractor’s  design  prior  to  Milestone  B.  The  program 
plans  to  continue  competition  throughout  engineering  and  manufacturing 
development.  None  of  the  four  programs  in  our  review  received  a  waiver 
from  Reform  Act  provisions. 


Additional  Emphasis  Is 
Being  Placed  on  Improving 
Milestone  Review  Process, 
Policies,  and  Service 
Capabilities 


OSD  is  taking  additional  steps  to  strengthen  the  department’s  oversight  of 
weapon  acquisition  programs  and  guidance  for  developing  the  programs. 
In  June  201 1 ,  for  example,  the  Under  Secretary  for  Acquisition, 
Technology  and  Logistics  revised  the  weapons  acquisition  review  process 
to  consider  acquired  knowledge  on  weapon  acquisition  programs  earlier 
than  before.  The  revised  review  process  includes  two  new  review  points. 
The  first  new  review — the  pre-engineering  and  manufacturing 
development  review — occurs  before  the  release  of  a  final  request  for 
proposal  for  the  engineering  and  manufacturing  development  phase.  The 
purpose  of  this  new  review  is  to  assess  each  program’s  acquisition 
strategy,  request  for  proposal,  and  key  related  planning  documents  earlier 
in  the  process,  and  to  determine  whether  program  plans  are  affordable, 
executable,  and  reflect  sound  business  arrangements.  The  second  new 
review — the  acquisition  strategy  and  request  for  proposals  review  and 
approval — occurs  prior  to  Milestone  C,  the  production  decision.  The 
review  provides  the  milestone  decision  authority  an  opportunity  to  review 
the  acquisition  strategy  and  request  for  proposals  for  the  production  and 
deployment  phase  prior  to  Milestone  C.  Figure  2  illustrates  the  revised 
review  process. 
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Figure  2:  Revised  Review  Process  for  Major  Defense  Acquisition  Programs 
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Source:  GAO  analysis  of  DOD  data. 


According  to  the  Under  Secretary  of  Defense  for  Acquisition,  Technology 
and  Logistics,  who  is  the  authority  for  making  milestone  decisions  for 
most  major  weapon  acquisitions,  the  prior  review  process  did  not  provide 
an  adequate  opportunity  for  review  of  program  plans  prior  to  release  of 
the  final  request  for  proposals — the  point  at  which  DOD’s  requirements, 
schedule,  planned  program  content,  and  available  funding  should  be  firm 
and  available  for  review.  Further,  the  Under  Secretary  stated  that  making 
changes  to  acquisition  strategies  and  program  plans  after  all  bidding 
activities,  proposal  evaluation,  and  source  selection  are  complete  is 
difficult  and  highly  disruptive.13 

DOD  is  also  rewriting  the  DOD  Instruction  5000. 0214  to  include  an 
extensive  restructure  of  acquisition  policies  according  to  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics.  This 
update  will  implement  Section  832  of  the  National  Defense  Authorization 
Act  for  Fiscal  Year  201 2, 15  which  requires  DOD  to  issue  guidance  on 
actions  to  be  taken  to  improve  its  processes  for  estimating,  managing, 
and  reducing  operation  and  support  cost,  as  well  as  ensure  competition  in 
maintenance  and  sustainment  of  subsystems  of  major  weapon  systems, 
among  other  things.  In  addition  to  current  policies  implementing  the 
Reform  Act,  officials  stated  that  key  provisions  from  the  Reform  Act  will 
also  be  included  in  the  updated  instruction. 


13  DOD  Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics,  “Improving  Milestone  Process  Effectiveness,”  (June  23,  2011). 

14  DOD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System,  Dec.  2,  2008. 

15  Pub.  L.  No.  112-81  §  832. 
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In  addition  to  implementing  the  provisions  of  the  Reform  Act,  DOD  offices 
have  taken  other  steps  to  strengthen  acquisition  capabilities  throughout 
the  department.  For  example: 


•  The  SE  office,  according  to  DOD  officials,  led  efforts  to  establish 
working  groups  to  help  the  services  address  systemic  reliability  issues 
across  the  unmanned  aircraft  and  rotary  wing  portfolios  earlier  in  the 
process.  The  office  also  led  several  workforce  development  initiatives 
to  attract  and  retain  a  qualified  engineering  workforce  and  supported 
the  implementation  of  legislation  requiring  each  acquisition  program 
office  to  name  a  key  technical  advisor  who  is  responsible  for  all 
engineering  activities. 

•  The  DT&E  office,  according  to  DOD  officials,  championed  updates  to 
DOD  Instructions  that  will  require  weapon  acquisition  programs  to 
consider  using  DOD  test  capabilities  before  paying  contractors  to 
develop  similar  capabilities.  In  addition,  the  office  supported 
legislation  requiring  major  defense  acquisition  program  offices  to  have 
a  government  test  agency  serving  as  the  lead  developmental  test  and 
evaluation  organization  for  the  program  and  a  chief  developmental 
tester.  The  chief  developmental  tester  position,  as  included  in  the 
National  Defense  Authorization  Act  for  Fiscal  Year  2012,  is  tasked 
with  coordinating  the  planning,  management,  and  oversight  of  all 
developmental  testing  activities,  among  other  things.16 

•  The  CAPE  office,  according  to  DOD  officials,  established  an  operating 
and  support  cost  directorate  to  build  its  expertise  and  place  more 
emphasis  on  developing  better  operating  and  support  cost  estimates 
throughout  the  acquisition  life  cycle.  This  directorate  will  coordinate 
the  development  of  an  operating  and  support  cost  estimating 
guidebook. 

•  The  PARCA  office,  according  to  DOD  officials,  is  providing  additional 
insights  to  the  Under  Secretary  of  Defense  for  Acquisition, 

Technology  and  Logistics  on  systemic  acquisition  problems. 
Specifically,  the  office  is  examining  a  wide  range  of  acquisition-related 
information  from  the  past  40  years,  such  as  contract  type,  stability  of 
key  performance  parameters,  and  program  manager  tenure  to 


16  Pub.  L.  No.  112-81  §  835. 


Page  13 


GAO-13-103  Weapons  Acquisition  Reform 


determine  if  there  is  any  statistical  correlation  between  these  factors 
and  good  or  poor  acquisition  outcomes. 


The  Reform  Act  Is 
Helping  Programs 
Identify  and  Mitigate 
Risks  Earlier  in  the 
Acquisition  Process 


We  identified  four  key  areas  where  the  Reform  Act  had  a  significant 
influence  on  programs  in  the  3  years  since  it  was  enacted: 

(1)  requirements,  (2)  cost  and  schedule,  (3)  testing,  and  (4)  reliability. 
These  four  areas  have  been  common  sources  of  problems  in  the  past. 
For  example,  the  services  typically  started  new  weapon  acquisition 
programs  with  requirements  that  were  both  demanding  and  inflexible  and 
planned  to  use  relatively  unproven  technologies  to  meet  the 
requirements — all  of  which  increased  program  risks.  In  addition,  cost  and 
schedule  estimates  were  frequently  too  optimistic  based  on  the  proposed 
requirements  and  technologies.  Design  problems  stemming  from  rigid 
requirements  and  the  use  of  immature  technologies  to  meet  them  were 
often  discovered  during  testing  and  fixed  late  in  the  development  cycle 
and  resulted  in  cost  increases,  performance  shortfalls,  and  schedule 
delays.  Finally,  DOD’s  inattention  to  reliability  has  resulted  in  a  dramatic 
increase  in  the  number  of  systems  that  have  not  met  suitability 
requirements  during  operational  testing.  Deficiencies — such  as  high 
failure  rates  and  disappointing  improvements  in  the  reliability,  availability, 
and  maintainability  of  weapon  systems — have  limited  program 
performance  and  increased  operation  and  support  costs.17  We  examined 
1 1  programs  at  various  stages  of  the  acquisition  process  to  determine 
how  the  offices  and  policies  established  as  a  result  of  the  Reform  Act 
impacted  their  acquisition  strategies  and  decision-making  process.  Four 
programs  had  not  yet  passed  Milestone  B,  development  start,  at  the  time 
we  began  our  review.  Of  the  remaining  seven  programs,  three  had 
breached  Nunn-McCurdy  cost  thresholds  since  the  act  was  passed  and 
have  had  to  satisfy  the  Reform  Act’s  new  requirements  with  regards  to 
certification.  The  other  programs  had  significant  interaction  with  one  or 
more  of  the  OSD  offices  established  by  the  Reform  Act.  Table  3  indicates 
in  which  of  the  four  areas  each  program  has  been  affected  by  the  Reform 
Act. 


17  DOD  Report  of  the  Defense  Science  Board  Task  Force  on  Developmental  Test  and 
Evaluation  (May  1, 2008). 
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Table  3:  Reform  Act  Influence  on  Case  Study  Programs 

Program 

Requirements  Cost  and  schedule 

Testing 

Reliability 

Before  Milestone  B 

Ground  Combat  Vehicle 

>/  ✓ 

•/ 

✓ 

Joint  Light  Tactical  Vehicle3 

>/  ✓ 

V 

V 

Ohio  Class  Replacement 

✓  >/ 

V 

V 

Ship  to  Shore  Connector3 

s 

V 

After  Milestone  B 

Joint  Strike  Fighter 

✓ 

Global  Hawk 

s 

✓ 

Gray  Eagle 

V 

KC-46  Tanker 

V 

s 

Littoral  Combat  Ship  Seaframe 

✓ 

Remote  Minehunting  System 

✓ 

s 

s 

Small  Diameter  Bomb  II 

Source:  GAO  analysis  of  DOD  data. 


aDuring  the  course  of  our  review,  the  Joint  Light  Tactical  Vehicle  and  Ship  to  Shore  Connector 
programs  held  a  Milestone  B  review. 


In  some  cases,  programs  have  made  changes  based  on  input  from  OSD 
offices  like  systems  engineering  or  developmental  test  and  evaluation;  in 
other  cases,  programs  have  integrated  Reform  Act  policies,  such  as 
preliminary  design  reviews  and  competitive  prototyping,  into  their 
acquisition  planning.  Programs  that  were  already  in  development  or 
production  when  the  Reform  Act  was  passed  were  less  likely  to  have 
interactions  with  the  OSD  offices  on  requirements  trades  because  these 
discussions  typically  occur  prior  to  Milestone  B.  A  discussion  of  how 
individual  programs  have  been  affected  in  the  areas  of  requirements,  cost 
and  schedule  realism,  testing,  and  reliability  follows. 


Greater  Emphasis  Is  Being  The  Reform  Act  places  significant  emphasis  on  early  problem  solving  and 
Placed  on  Requirements,  requires  programs  to  put  much  more  effort  toward  considering  trade-offs 
but  Challenges  Remain  among  cost,  schedule,  and  performance  requirements  prior  to  Milestone 

. 1  B.  As  part  of  this  effort,  it  requires  the  Secretary  of  Defense  to  ensure  that 

acquisition,  budget,  and  cost  estimating  officials  have  the  opportunity  to 
raise  cost  and  schedule  matters  before  performance  objectives  are 
established.  The  Reform  Act  also  charges  the  Joint  Requirements 
Oversight  Council  with  the  responsibility  to  ensure  that  cost,  schedule, 
and  performance  trade-offs  for  joint  military  requirements  are  considered, 
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and  to  include  combatant  commanders  in  the  process  to  ensure  the 
user’s  needs  are  adequately  satisfied.  The  offices  established  as  a  result 
of  the  Reform  Act  have  helped  programs,  such  as  the  Joint  Light  Tactical 
Vehicle  and  Ground  Combat  Vehicle,  make  trade-offs  among  cost, 
schedule,  and  technical  performance  requirements.  As  a  result,  these 
programs  have  developed  a  more  realistic  acquisition  strategy  from  a 
cost,  schedule,  and  technical  standpoint. 

•  Joint  Light  Tactical  Vehicle:  The  program  held  several  reviews  prior 
to  Milestone  B  to  identify,  modify,  or  eliminate  requirements  that  were 
unachievable  or  unaffordable,  thus  establishing  a  more  technically 
realistic  program.  Officials  from  DT&E  and  SE  participated  in  these 
reviews.  By  involving  both  the  requirements  and  acquisition 
communities  in  the  reviews,  the  Army  was  able  to  reduce  the  required 
capability  to  cut  costs  while  ensuring  that  trade-off  decisions  would 
not  impair  the  system’s  ability  to  meet  the  warfighter’s  operational 
needs.  Examples  of  requirements  changes  that  helped  to  cut  costs  as 
well  as  reduce  risk  include: 

•  allowing  the  active  suspension  system,  crew  displays,  and 
integrated  starter-generator  to  be  tradable  design  features.  These 
changes  resulted  in  a  30  percent  reduction  in  the  average  unit 
manufacturing  cost  from  the  initial  target  of  $475,000  to  $331,000, 
while  at  the  same  time  reducing  technical  and  weight  risk. 
According  to  program  officials,  this  makes  the  $250,000  unit 
manufacturing  goal  more  achievable. 

•  reducing  the  reliability  requirement  and  changing  the  Army 
helicopter  lift  requirement  based  on  the  results  of  technology 
development  prototype  testing.  This  mitigated  technical  risks 
going  into  development. 

The  program  recently  moved  into  engineering  and  manufacturing 
development,  but  not  all  requirements  issues  have  been  resolved  and 
future  trade-offs  may  be  necessary.  For  example,  early  testing 
showed  that  none  of  the  three  prototype  variants  met  the  program’s 
soft  soil  or  sand  slope  requirement.  This  requirement  has  not  been 
changed.  Program  and  OSD  officials  are  monitoring  this  issue  closely 
and  plan  to  actively  manage  it  during  engineering  and  manufacturing 
development. 

•  Ground  Combat  Vehicle:  The  Ground  Combat  Vehicle  program 
exhibited  some  of  the  same  problems  experienced  by  previous  DOD 
programs  prior  to  Milestone  B — demanding  and  inflexible 
requirements.  The  SE  office  and  the  Under  Secretary  of  Defense  for 
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Acquisition,  Technology  and  Logistics  are  helping  the  program  set 
achievable  requirements.  Following  its  materiel  development  decision 
in  February  2010,  the  program  issued  a  request  for  proposals  that 
contained  nearly  1,000  requirements  and  a  challenging  7-year 
schedule  for  the  delivery  of  the  first  production  vehicle.  At  the  request 
of  the  Under  Secretary  for  Acquisition,  Technology  and  Logistics,  the 
Army  established  an  independent  review  team  to  assess  the  risks 
associated  with  the  program’s  schedule.  The  team,  which  included  an 
SE  official,  raised  concerns  about  the  program’s  high  number  of 
mandatory  requirements  and  the  risks  associated  with  the  7-year 
schedule.  To  mitigate  program  risks,  the  Army  reduced  the  number  of 
performance  requirements  by  about  25  percent  and  prioritized  the 
others,  giving  competing  contractors  flexibility  in  addressing  some 
performance  requirements.  The  Army  issued  a  revised  request  for 
proposal  in  November  201 0.  In  August  2011,  the  Under  Secretary  of 
Acquisition,  Technology  and  Logistics  approved  the  program’s  entry 
into  technology  development,  but  expressed  concern  about  the  cost 
and  schedule  risks  associated  with  delivering  a  production  vehicle  in  7 
years.  Because  of  these  concerns,  the  Under  Secretary  directed  the 
Army  to  consider  other  alternatives,  such  as  existing  vehicles,  that 
could  meet  warfighter  needs.  The  analysis  is  currently  planned  to  be 
completed  in  March  2013  to  inform  the  Milestone  B  decision. 


Program  Cost  and 
Schedule  Estimates  Are 
Becoming  More  Realistic 


By  establishing  a  new  cost  assessment  and  program  evaluation  office 
and  requiring  this  office  to  scrutinize  program  cost  and  schedule 
estimates  beginning  at  Milestone  A,  CAPE  officials  believe  that  the 
Reform  Act  has  helped  infuse  more  realism  in  cost  estimates  and 
promote  earlier  discussions  about  affordability.  CAPE  officials  also 
believe  that  because  their  independent  cost  and  schedule  estimates  have 
become  more  visible  within  DOD  and  Congress,  the  military  services  are 
developing  more  realistic  estimates.  We  saw  evidence  of  these  benefits  in 
the  programs  we  reviewed,  including  the  Ohio  Class  Replacement, 

Littoral  Combat  Ship  Seaframe,  and  F-35  Joint  Strike  Fighter  programs. 


•  Ohio  Class  Replacement:  The  CAPE  office  was  involved  in  the 
decision-making  process  to  ensure  program  affordability.  The  office 
prepared  an  independent  cost  estimate  and  reviewed  the  program’s 
affordability  goals  prior  to  Milestone  A.  The  service  and  independent 
estimates  were  within  2  percent  of  each  other.  However,  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics 
directed  the  Navy  to  do  a  rigorous  cost  comparison  of  a  16  missile 
tube  design  versus  a  20  missile  tube  design.  The  Navy  determined 
that  a  16  missile  tube  configuration  would  meet  warfighter 
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requirements  and  users’  needs  while  reducing  program  costs  by 
about  $200  million  per  submarine,  or  approximately  $3  billion  for  the 
total  program.  It  would  also  simplify  the  ships’  design  and  integration 
effort.  The  CAPE  office  validated  the  savings  associated  with  the  16 
missile  tube  design.  As  a  result,  the  Navy  incorporated  the  16  missile 
configuration  as  the  program  baseline. 

•  Littoral  Combat  Ship  Seaframe:  The  CAPE  office  helped  make 
program  costs  more  visible.  Prior  to  the  program’s  Milestone  B 
decision,  CAPE  completed  an  independent  cost  estimate  of  the 
seaframe  program  and  found  that  the  resources  in  the  future  years’ 
defense  plan  budget  were  lower  than  the  projected  program  costs  for 
the  same  time  period.  Navy  officials  attributed  this  problem  to  the 
overlap  between  the  timing  of  the  milestone  decision  and  the 
president’s  budget  submission.  The  office  further  noted  that  the 
resources  in  the  Navy’s  budget  did  not  include  the  additional 
development  activities  required  to  support  two  full  ships.  Without  this 
information,  decision  makers  would  not  have  had  visibility  into  the 
expected  costs  of  the  seaframe  program  or  be  able  to  make  more  fully 
informed  decisions.  As  a  result,  the  Navy  re-phased  its  funding  in  the 
budget,  adding  approximately  $397  million  to  fully  fund  the 
development  program. 

Joint  Strike  Fighter:  The  SE  and  CAPE  offices  helped  the  program 
develop  more  realistic  cost  and  schedule  estimates.  CAPE  officials 
have  been  involved  in  reviews  of  the  F-35  Joint  Strike  Fighter  program 
even  prior  to  the  passage  of  the  Reform  Act  and  have  continued  to  be 
heavily  involved  in  subsequent  programs  reviews.  For  example,  the 
cost  analysis  improvement  group,  which  was  the  predecessor  to 
CAPE,  led  a  multi-functional  joint  estimating  team  review  of  the 
program  in  2008.  This  review  found  problems  with  the  program’s 
funding  and  schedule.  In  2010,  the  Joint  Strike  Fighter  program 
notified  Congress  that  its  estimated  unit  costs  had  increased  by  more 
than  80  percent  since  the  original  Milestone  B  baseline  in  2001 .  This 
increase  triggered  a  Nunn-McCurdy  unit  cost  breach  and  later 
prompted  the  program  executive  officer  to  commission  a  technical 
baseline  review  of  the  program  to  help  determine  the  resources 
needed  to  complete  development.  Officials  from  the  SE  office 
participated  in  this  technical  review.  The  CAPE  office  also  did  an 
independent  cost  estimate  of  the  program  as  part  of  the  Nunn- 
McCurdy  certification  process.  Based  on  information  from  these 
efforts,  DOD  developed  a  more  realistic  program  plan  by  adding  $4.6 
billion  to  the  development  program,  reducing  near-term  procurement 
quantities  by  125  aircraft,  and  extending  the  development  test  period 
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by  4  years  to  accommodate  developmental  testing,  address  the 
increased  program  scope,  and  fix  software  issues. 


Programs  Are  Adding  More 
Tests  to  Reduce 
Development  Risks 


The  Reform  Act  significantly  strengthened  the  role  of  developmental 
testing  in  the  department.  In  the  2  decades  prior  to  the  Reform  Act,  the 
prominence  of  developmental  testing  had  declined  within  OSD.  In  the 
early  1990’s,  developmental  testing  was  part  of  an  all-encompassing  test 
organization  that  reported  directly  to  the  Under  Secretary  of  Defense  for 
Acquisition.  According  to  a  former  senior  developmental  testing  official,  by 
2004,  two  people  worked  on  developmental  testing  activities  within  the 
systems  engineering  organization.  In  establishing  a  separate  office  for 
DT&E,  the  Reform  Act  reinforced  the  need  for  robust  developmental 
testing  early  in  the  acquisition  process.  The  Reform  Act  gave  the  Deputy 
Assistant  Secretary  for  Developmental  Test  and  Evaluation  formal 
approval  authority  for  the  test  and  evaluation  master  plans  of  major 
defense  acquisition  programs.  This  authority  enables  the  Deputy 
Assistant  Secretary  to  help  ensure  that  programs  have  robust  test  and 
evaluation  plans.  Our  case  study  reviews  illustrate  the  efforts  that  the 
DT&E  office  has  made  to  help  programs  such  as  the  Small  Diameter 
Bomb  II  and  KC-46  Tanker  obtain  more  design  and  performance 
knowledge  early  in  a  program’s  acquisition  life  cycle. 


•  Small  Diameter  Bomb  II:  Air  Force  program  officials  acknowledged 
that  developmental  and  operational  testing  officials  worked  closely 
with  them  as  they  prepared  the  test  and  evaluation  master  plan  for  the 
Milestone  B  decision.  After  reviewing  the  plan,  developmental  and 
operational  test  officials  concluded  that  the  program  would  benefit 
from  adding  a  28-shot  test  program  prior  to  entering  operational 
testing.  According  to  the  program  office,  the  purpose  of  the  additional 
testing  is  to  further  establish  the  performance  of  the  weapon  in 
realistic  scenarios  and  to  increase  the  likelihood  of  completing 
operational  testing  without  a  failure.  DT&E  officials  stated  this  testing 
would  provide  more  complete  knowledge  about  the  bomb’s 
functionality  and  help  reduce  risk  of  a  major  redesign  moving  forward. 
Program  officials  stated  that  they  allocated  an  additional  $41  million  to 
its  developmental  test  program  to  conduct  the  28  additional  flight  tests 
prior  to  operational  testing. 


•  KC-46  Tanker:  The  program  office  acknowledged  that  the  DT&E 

office,  as  part  of  an  integrated  test  team  comprised  of  government 
and  industry  officials,  helped  identify  options  that  could  add  time  to  the 
test  plan  for  important  testing  if  unexpected  delays  are  encountered. 

In  its  fiscal  year  201 1  annual  report,  the  operational  test  and 
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evaluation  office  reported  that  the  KC-46’s  planned  flight  test  program 
was  not  executable,  determining  that  more  time  would  be  needed  for 
military  flight-testing.  It  based  this  conclusion  on  the  historical  flight 
test  experience  of  similar  programs.  Program  officials  stated  that  they 
were  initially  reluctant  to  change  the  test  plan  because  they  had 
awarded  a  fixed  price  contract  and  any  changes  could  result  in 
reopening  the  contract,  leading  to  potential  cost  increases.  However, 
the  integrated  test  team  identified  a  recovery  period  that  may  be 
applied  to  the  KC-46  aerial  refueling  certification  if  delays  are 
encountered.  The  contractor  now  has  a  plan  that  could  allocate  an 
additional  1 .5  months  for  two  test  aircraft  to  complete  this  testing,  if 
deemed  necessary.  This  testing  would  provide  more  knowledge  about 
the  program’s  aerial  refueling  performance  prior  to  operational  testing. 
DT&E  officials  stated  that  they  plan  to  continue  working  with  the 
program  to  address  overall  flight  test  challenges.  While  testing 
remains  one  of  the  program’s  risk  areas,  this  change  may  lessen  that 
risk. 


Greater  Emphasis  Is  Being 
Placed  on  Weapon  System 
Reliability 


The  Reform  Act  emphasizes  the  need  for  designing  more  reliable  weapon 
systems.  It  charges  the  Deputy  Assistant  Secretary  for  Systems 
Engineering  with  the  responsibility  to  ensure  the  systems  engineering 
approach  used  by  major  acquisition  programs  includes  a  robust  plan  for 
improving  reliability.  The  DT&E  office  reviews  programs’  reliability  growth 
test  plans.  This  testing  provides  visibility  over  how  reliability  is  improving 
and  uncovers  design  problems  so  fixes  can  be  incorporated  before 
production  begins.  A  reliability  growth  curve  is  used  to  track  projected  and 
actual  improvements  in  reliability  over  time.  The  Reform  Act  further 
requires  that  the  Deputy  Assistant  Secretary  for  Systems  Engineering 
develop  policies  and  guidance  for  the  inclusion  of  provisions  relating  to 
systems  engineering  and  reliability  growth  in  requests  for  proposals.  We 
observed  evidence  of  this  increased  emphasis  in  the  Joint  Light  Tactical 
Vehicle,  Remote  Minehunting  System,  Gray  Eagle,  and  Global  Hawk 
programs. 


•  Joint  Light  Tactical  Vehicle:  The  DT&E  office  helped  this  program 
develop  a  more  realistic  reliability  growth  plan  prior  to  Milestone  B. 
Based  on  the  performance  of  prototype  vehicles,  developmental  test 
officials  determined  that  the  program’s  reliability  growth  curve  was 
unrealistic.  For  example,  officials  reported  that  the  program’s  initial 
reliability  growth  plan  assumed  a  starting  reliability  that  was  almost  60 
percent  higher  than  what  had  actually  been  demonstrated  during 
technology  development.  It  also  assumed  commonality  between  the 
two  vehicle  variants,  a  large  reliability  increase  in  a  short  test  time, 
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and  two  corrective  action  periods.  The  DT&E  office  recommended 
that  the  program  eliminate  the  vehicle  commonality  assumption,  add 
more  test  miles,  and  add  another  corrective  action  period  to  its  test 
plan.  It  also  recommended  that  the  program  consider  lowering  the 
vehicle’s  reliability  requirement.  Based  on  this  input,  the  Army  revised 
its  plan  by  adding  two  vehicles  and  40,000  more  test  miles  to  ensure 
reliability  is  adequately  addressed  for  both  variants.  With  approval  of 
the  user,  it  also  reduced  the  reliability  requirement  from  3,600  to  2,400 
miles  mean  time  between  operational  mission  failures. 

•  Remote  Minehunting  System:  The  SE  office  worked  with  program 
officials  to  improve  reliability  growth  planning,  which  was  found  to  be 
one  of  the  key  factors  leading  to  the  program’s  Nunn  McCurdy  unit 
cost  breach  in  2009.  Before  the  breach,  program  officials  had  not 
funded  a  reliability  growth  program  or  established  a  design  for 
reliability  process.  The  program  had  a  reliability  goal  of  150  hours 
mean  time  between  failures,  but  program  officials  stated  that  testing 
demonstrated  a  reliability  of  only  about  45  hours.  Since  the  breach, 
the  program  has  worked  closely  with  the  SE  office  to  establish  a 
reliability  program  plan  and  a  growth  curve  to  track  reliability 
improvements.  During  the  Nunn-McCurdy  review,  the  program 
developed  a  three  phase  reliability  growth  program  to  improve  the 
program’s  subsystems,  components,  and  manufacturing  processes 
that  contributed  to  poor  reliability.  According  to  program  officials, 
phase  one  of  the  reliability  growth  program  was  completed  in  201 1 , 
and  reliability  has  improved  by  40  percent,  going  from  45  hours  mean 
time  between  operational  mission  failures  to  63  hours.  Although  this 
improvement  is  still  below  the  minimum  requirement  of  75  hours, 
program  officials  stated  that  phase  two  of  the  reliability  growth 
program  is  scheduled  to  be  completed  in  April  2013  and  is  projected 
to  achieve  the  program’s  75  hour  minimum  requirement. 

•  Gray  Eagle  and  Global  Hawk:  The  SE  office  has  worked  to  improve 
reliability  across  the  unmanned  aircraft  portfolio,  including  the  Gray 
Eagle  and  Global  Hawk.  Prior  to  the  Gray  Eagle’s  second  low  rate 
initial  production  decision  in  201 1,18  SE  officials  raised  concerns  about 
the  system’s  poor  reliability.  As  a  result,  the  Army  was  directed  to 
undertake  a  reliability  improvement  program.  The  Under  Secretary  of 
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Low-rate  initial  production  is  production  of  the  system  in  the  minimum  quantity 
necessary  to  provide  articles  for  operational  tests,  establish  an  initial  production  base,  and 
permit  an  orderly  increase  in  the  production  rate  for  the  system. 
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Defense  for  Acquisition,  Technology  and  Logistics  approved  the 
program  for  low  rate  initial  production,  but  stressed  the  need  to 
improve  the  operational  reliability  as  quickly  as  possible.  SE  officials 
worked  with  program  officials  to  establish  a  reliability  working  group, 
develop  reliability  growth  curves,  and  develop  a  reliability 
enhancement  management  plan.  According  to  SE  officials,  the  Gray 
Eagle  initially  improved  the  reliability  of  the  aircraft  by  15  percent  and 
the  ground  control  station  by  30  percent.  According  to  program 
officials,  the  initial  reliability  goals  were  overstated  and  not  needed  to 
meet  the  program’s  overall  operational  availability  requirement.  Based 
on  initial  operational  test  results  in  August  2012,  the  program  office  is 
working  with  the  user  to  redefine  the  reliability  goals  without  impacting 
the  system’s  ability  to  meet  its  overall  operational  availability 
requirement.  According  to  the  PARCA  office,  these  efforts  have  been 
informed  by  a  detailed  reliability  model  that  they  built  in  consultation 
with  the  Army.  This  model  showed  the  relationship  between  the 
aircraft’s  reliability  and  its  availability  to  perform  operational  missions. 
SE  officials  also  found  similar  reliability  problems  on  the  Global  Hawk 
program  and  worked  with  program  officials  to  establish  a  reliability 
growth  and  improvement  plan  and  reliability  growth  curves.  According 
to  SE  officials,  the  time  between  unscheduled  maintenance  on  the 
Global  Hawk  has  improved  on  the  order  of  50  to  80  percent. 


Challenges  Exist  That 
Could  Limit  the 
Reform  Act’s  Ability 
to  Influence  Systemic 
Change 


While  DOD  has  taken  steps  to  implement  most  of  the  fundamental 
Reform  Act  provisions,  some  key  efforts  to  date  have  been  primarily 
focused  on  DOD’s  largest  major  defense  acquisition  programs. 

Expanding  the  reach  of  the  Reform  Act  to  bring  about  systemic  change  to 
DOD’s  weapons  acquisition  process  so  that  it  influences  all  programs, 
however,  still  has  challenges.  Although  senior  leaders  were  receptive  to 
the  Reform  Act  principles,  they  identified  several  challenges  that  currently 
limit  DOD’s  ability  to  broaden  the  Reform  Act’s  influence.  We  grouped 
these  challenges  into  five  general  categories:  (1)  organizational  capability 
constraints;  (2)  need  for  additional  guidance  on  cost  estimating  and 
Reform  Act  implementation;  (3)  uncertainty  about  the  sufficiency  of 
service  level  systems  engineering  and  developmental  testing  resources; 
(4)  limited  dissemination  of  lessons  learned;  and  (5)  cultural  barriers. 


Organizational  Capability 
Constraints  Make  It 
Difficult  to  Expand  Reform 
Act’s  Impact 


Leaders  of  two  of  the  offices  established  as  a  result  of  the  Reform  Act  told 
us  that  even  though  they  have  implemented  most  of  the  fundamental 
Reform  Act  provisions,  they  have  had  to  limit  their  activities  to  a  portion  of 
acquisition  programs  in  their  portfolios  due  to  resource  constraints.  Thus, 
it  is  doubtful  that  they  could  expand  the  scope  of  their  activities  to  include 
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more  weapon  acquisition  programs  at  current  staffing  levels.  For 
example,  the  DT&E  office  has  had  to  be  selective  in  its  level  of  oversight 
of  acquisition  programs  because  the  current  staff  of  around  70 
government  and  contractor  personnel  cannot  adequately  cover  a  portfolio 
of  over  200  acquisition  programs,  according  to  its  Deputy  Assistant 
Secretary.  The  office  has  dropped  virtually  all  but  the  largest  programs 
from  its  oversight  list  and  eliminated  oversight  of  some  major  automated 
information  systems.  CAPE  officials  estimated  that  its  cost  assessment 
division  would  need  to  double  in  size  in  order  to  meet  the  Reform  Act’s 
requirements.  However,  soon  after  the  Reform  Act  was  enacted, 
budgetary  constraints  limited  the  expansion  of  the  cost  estimating 
workforce  to  about  25  percent  of  the  necessary  growth.  According  to 
CAPE  officials,  its  current  cost  analysis  staffing  is  not  adequate  to  meet 
its  mission  of  improving  the  analytical  skills  of  the  defense  cost  estimating 
workforce,  issuing  policy,  and  providing  sound  and  unbiased  cost  and 
schedule  estimates.  The  office  has  delegated  its  independent  cost 
estimating  responsibility  for  most  major  automated  information  systems  to 
the  military  services  and  some  guidance  has  yet  to  be  issued. 

The  SE  and  PARCA  offices  are  also  struggling  in  some  regards.  For 
example,  according  to  its  Deputy  Assistant  Secretary,  the  SE  office  is 
continuously  challenged  to  maintain  the  high  caliber,  qualified  personnel 
required  to  provide  assistance  to  and  oversight  of  its  portfolio  of  over  200 
acquisition  programs.  Further,  PARCA  officials  stated  that  the  availability 
of  government  positions,  particularly  at  the  senior  executive  service  level, 
continues  to  be  a  critical  issue  for  the  office.  The  two  divisions  within  the 
PARCA  office,  the  performance  assessments  division  and  the  root  cause 
division,  do  not  currently  have  permanent  government  personnel  at  the 
senior  executive  level.  Officials  also  stated  current  proprietary  information 
rules  limit  the  ability  of  PARCA  contractor  personnel  to  handle  and 
maintain  some  weapon  system  information,  severely  impeding 
operations. 


Services  Would  Like  More 
Cost  Estimating  and 
Reform  Act 

Implementation  Guidance 


Offices  within  OSD  have  not  yet  issued  more  detailed  guidance  that  could 
help  institutionalize  better  cost  estimating  practices  and  steer  program 
decisions  related  to  competitive  prototyping  and  preliminary  design 
reviews.  The  CAPE  office  has  not  issued  guidance  for  operating  and 
support  costs  estimates,  such  as  fuel  and  maintenance  costs,  that  have 
been  estimated  to  account  for  two-thirds  or  more  of  a  system’s  total  life 
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cycle  cost.19  In  addition,  although  not  specifically  required  by  the  Reform 
Act,  the  CAPE  office  has  not  issued  guidance  for  the  services  to  use 
when  developing  Milestone  A  program  cost  estimates.  As  a  result,  senior 
leaders  may  not  have  access  to  realistic  cost  estimates  prior  to  Milestone 
B  for  decision  making  purposes.  Military  service  officials  told  us  they  are 
particularly  interested  in  getting  guidance  on  data  that  should  be  included 
in  the  cost  analysis  requirements  description,  which  forms  the  basis  for 
their  cost  estimates  at  Milestones  B  and  C.  CAPE  officials  recognize  that, 
while  some  progress  has  been  made,  they  need  to  complete  the 
guidance,  but  have  not  been  able  to  dedicate  resources  to  do  so. 

Some  officials  also  told  us  that  they  find  the  competitive  prototyping  and 
preliminary  design  review  requirements  confusing  and  would  like 
guidance  on  how  to  implement  these  requirements.  DOD  policy  requires 
the  technology  development  strategy  for  major  defense  acquisition 
programs  to  provide  for  prototypes  of  the  system  or,  if  a  system  prototype 
is  not  feasible,  for  prototypes  of  critical  subsystems  before  Milestone  B 
approval.20  However,  officials  from  the  Ground  Combat  Vehicle  program 
were  unclear  as  to  when  and  what  type  of  prototype  to  use.  From  a 
broader  perspective,  other  military  officials  questioned  the  value  of 
competitive  prototyping  as  a  blanket  requirement  for  all  programs, 
especially  for  programs  that  are  using  mature  technologies,  given  the 
cost.  For  example,  senior  acquisition  officials  questioned  the  necessity  of 
spending  $400  million  on  competitive  prototyping  for  the  Small  Diameter 
Bomb  II  program  since  the  program  was  aware  of  problems  with  one 
contractor’s  design.  However,  program  officials  indicated  that  competitive 
prototyping  enabled  them  to  identify  design  issues  early  in  development 
and  realize  a  savings  of  $1  billion.  Officials  from  the  Ground  Combat 
Vehicle  program  we  spoke  with  also  indicated  that  they  struggled  with  the 
timing  of  when  to  hold  the  program’s  preliminary  design  review  and  what 
type  of  knowledge  was  required,  since  better  guidance  is  needed.  The 
program  plans  to  hold  multiple  design  reviews  prior  to  Milestone  B  to 
consider  contractor  and  government  designs  of  the  weapon  system  and 
then  hold  another  review  after  Milestone  B  in  order  to  resolve  differences 
between  the  government’s  and  selected  contractor’s  preliminary  designs. 


1 Q 

The  Reform  Act  required  CAPE  to  issue  guidance  relating  to  full  consideration  of  life 
cycle  management  and  sustainability  costs  in  major  defense  acquisition  programs. 

20  DOD  Directive-Type  Memorandum  09-027,  Implementation  of  the  Weapon  Systems 
Acquisition  Reform  Act  of  2009  (Dec.  4,  2009,  incorporating  Change  3,  Dec.  9,  201 1 ). 
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We  spoke  with  OSD  officials  to  determine  which  office  should  be 
providing  guidance  or  assistance  to  program  managers  on  competitive 
prototyping  and  preliminary  design  review  issues.  None  of  the  offices 
have  official  responsibility  for  these  efforts.  OSD  officials  stated  that  these 
are  program  decisions  and  should  be  discussed  with  their  respective 
military  service  level  acquisition  officials. 


Services  May  Not  Have 
Sufficient  Resources  to 
Oversee  and  Conduct 
Systems  Engineering  and 
Developmental  Testing 
Activities 


OSD  officials  believe  that  the  services  may  lack  resources  in  key 
positions  that  could  help  strengthen  systems  engineering  and 
developmental  testing  activities  on  weapon  acquisition  programs.  For 
example,  according  to  the  Deputy  Assistant  Secretary  of  Defense  for 
Systems  Engineering,  the  Navy  and  Air  Force  have  reassigned  the  duties 
and  responsibilities  of  their  service-level  chief  engineers,  thereby  de¬ 
emphasizing  the  importance  of  systems  engineering.  The  Deputy 
Assistant  Secretary  believes  maintaining  strong  systems  engineering 
leadership  at  the  service  level  is  essential  for  tying  the  systems 
engineering  community  together  and  promoting  good  systems 
engineering  practices  throughout  each  respective  service.  According  to 
the  DT&E  and  SE  office’s  March  2012  joint  annual  report  to  the 
Congress,  the  Navy  abolished  its  chief  engineer  position  and  while  the  Air 
Force  recently  began  to  take  steps  to  relocate  the  systems  engineering 
function  to  the  headquarters  level,  the  impact  of  a  recent  reorganization 
on  systems  engineering  activities  is  not  yet  known.  In  addition,  the  Deputy 
Assistant  Secretary  for  Developmental  Test  and  Evaluation  expressed 
concern  that  the  military  services  may  not  be  implementing  new 
legislation  that  requires  each  major  defense  acquisition  program  be 
supported  by  a  chief  developmental  tester  that  oversees  developmental 
test  and  evaluation  activities.  He  stated  that  in  some  cases  one  person  is 
serving  as  the  chief  developmental  tester  across  multiple  programs 
instead  of  having  one  person  dedicated  specifically  to  each  program.21 
The  Deputy  Assistant  Secretary  is  trying  to  determine  the  extent  to  which 
this  practice  is  occurring  and  then  plans  to  work  with  the  services  to  get 
more  focused  leadership  for  each  program. 


21  The  requirement  contained  in  the  National  Defense  Authorization  Act  for  Fiscal  Year 
2012  required  “each  major  defense  acquisition  program  to  be  supported  by  a  chief 
developmental  tester.”  However,  it  did  not  specifically  prohibit  one  person  serving  as  the 
chief  developmental  tester  across  multiple  programs. 
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It  is  also  unclear  whether  the  services  have  a  sufficient  number  of 
qualified  personnel  to  conduct  systems  engineering  and  test  and 
evaluation  activities.  The  services  planned  to  grow  these  workforces  by 
over  a  combined  5,000  people  between  fiscal  years  2009  and  2015  and 
had  made  progress  in  growing  each  of  these  workforces  through  fiscal 
year  2010.  However,  budget  cuts  have  resulted  in  DOD  canceling  some 
of  its  weapon  acquisition  programs  and  reassessing  its  decision  to 
increase  the  acquisition  workforce.  Last  year,  we  recommended  that  the 
Secretary  of  Defense  report  the  impact  budget  cuts  were  having  on  the 
military  service  workforce  and  their  ability  to  meet  weapon  acquisition 
program  needs  in  the  areas  of  developmental  testing  and  systems 
engineering.22  In  the  DT&E  and  SE  offices’  March  2012  joint  annual  report 
to  the  Congress,  the  Deputy  Assistant  Secretary  for  Systems  Engineering 
reported  that  the  Army  has  reduced  its  systems  engineering  workforce 
growth  plan  as  compared  to  the  plan  reported  in  March  201 1  joint  annual 
report,  and  that  contractor-to-civilian  conversions  have  been  suspended. 

In  addition,  the  Deputy  Assistant  Secretary  believes  a  prolonged  hiring 
freeze  in  the  Air  Force  could  potentially  create  new  experience  gaps  in 
the  workforce.  The  Deputy  Assistant  Secretary  for  Developmental  Test 
and  Evaluation  did  not  discuss  the  impact  of  budget  cuts  on  the  services’ 
test  and  evaluation  workforce  growth  plan  in  the  March  2012  joint  annual 
report  to  the  Congress  and  neither  office  reported  on  whether  the 
services  had  an  adequate  workforce  to  meet  the  needs  of  the  current 
portfolio  of  weapon  acquisition  programs. 


Expanded  Use  of  Lessons 
Learned  Would  Help 
Expand  Impact  across  the 
Acquisition  Portfolio 


DOD  has  not  taken  full  advantage  of  sharing  lessons  learned  obtained 
through  root  cause  analyses  of  programs  that  experience  Nunn-McCurdy 
cost  and  schedule  breaches  with  the  acquisition  workforce,  particularly 
program  managers.  According  to  the  Defense  Acquisition  Guidebook, 
which  provides  best  practices  the  acquisition  workforce  can  use  on 
programs,  lessons  learned  are  a  tool  that  the  program  manager  may  use 
to  help  identify  potential  areas  of  risk  associated  with  a  weapon 
acquisition  system  by  reviewing  the  experiences  encountered  in  past 
programs.  Lessons  learned  databases  document  what  worked  and  what 
did  not  work  in  past  programs,  in  the  hopes  that  future  programs  can 
avoid  the  same  pitfalls.  Further,  if  the  right  best  practices  are  applied, 
they  help  to  avoid  common  problems  and  improve  quality,  cost,  or  both. 


22  GAO-1 1-806. 
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The  PARCA  office  has  made  some  effort  to  educate  program  managers 
on  how  to  avoid  acquisition  problems  through  classes  taught  at  the 
Defense  Acquisition  University.  However,  these  courses  are  geared 
towards  educating  new  program  managers  and  may  not  be  reaching  a 
wide  range  of  program  officials.  Nevertheless,  some  officials  indicated 
that  this  information  would  be  helpful  for  program  officials  to  understand 
and  avoid  problems  that  have  affected  weapon  acquisition  programs  in 
the  past.  Other  officials  also  stated  that  it  would  be  helpful  if  root  cause 
analysis  assessments  contained  more  detailed  information  so  acquisition 
officials  could  better  understand  problems  and  apply  lessons  learned.  For 
example,  when  cost  estimating  was  determined  to  be  a  root  cause  of  a 
problem,  officials  stated  they  would  have  found  it  more  beneficial  to  know 
if  immature  technologies  or  unrealistic  requirements  were  the  basis  for 
the  poor  cost  estimate. 


Systemic  Changes  Will  Be  Perhaps  the  most  difficult  challenge  the  department  faces  in  making 

Difficult  Until  Cultural  systemic  changes  to  the  acquisition  process  is  changing  the  cultural 

Barriers  Are  Addressed  relationship  between  the  military  services,  which  fund  and  develop  new 
‘  1  '  weapon  acquisition  programs,  and  OSD  offices,  which  provide  advice  to 

and  oversee  the  programs.  Senior  military  service  officials  have  told  us 
they  believe  they  understand  and  can  manage  the  risks  of  specific 
weapon  acquisition  programs  without  much  assistance  from  OSD.  On  the 
other  hand,  OSD  officials  believe  more  assistance  is  needed,  as 
evidenced  by  the  high  number  of  programs  that  have  experienced  Nunn- 
McCurdy  breaches  and  poor  operational  testing  results.  For  example, 
since  it  was  established  in  2009,  the  DT&E  office  has  assessed  whether 
15  programs  were  ready  to  begin  operational  testing.  The  office 
recommended  that  5  of  the  programs — Global  Hawk  Block  20/30, 
Standard  Missile  6,  Joint  Tactical  Radio  System  Handheld  Manpack 
Small  Form  (HMS)  Rifleman  Radio,  Joint  Tactical  Radio  System  HMS 
Manpack,  and  MQ-1C  Gray  Eagle — not  proceed  into  operational  testing. 
However,  military  service  acquisition  chiefs  decided  to  allow  all  5  of  these 
programs  to  proceed  anyway.  Four  of  the  programs — Global  Hawk  Block 
20/30,  Standard  Missile  6,  Joint  Tactical  Radio  System  HMS  Rifleman 
Radio  and  Joint  Tactical  Radio  System  HMS  Manpack — demonstrated 
poor  performance  in  operational  testing,  in  areas  such  as  reliability, 
effectiveness,  or  suitability.  Operational  testing  results  for  the  MQ-1C 
Gray  Eagle  have  not  yet  been  reported. 

On  the  other  hand,  a  few  service  officials  we  met  with  were  reluctant  to 
accept  some  recommendations  made  by  OSD  offices  because  they 
believed  the  recommendations  were  overly  burdensome  and  could 
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significantly  impact  weapon  acquisition  programs’  cost  and  schedule 
outcomes  without  a  lot  of  benefit.  This  was  the  case  for  the  KC-46  Tanker 
program,  where  program  officials  were  concerned  that  additional  testing 
recommended  by  the  developmental  test  and  evaluation,  and  operational 
testing  offices,  as  part  of  the  integrated  test  team,  could  have  significant 
contractual  implications  during  development.  In  this  case,  officials 
identified  additional  flight  test  opportunities  without  having  to  renegotiate 
the  fixed  price  contract.  However,  the  additional  allotted  test  time  is  not 
equivalent  to  the  6  to  8  months  the  developmental  testing  office  felt 
should  be  added.  A  similar  situation  occurred  on  the  Ship  to  Shore 
Connector  program.  The  Navy  disagreed  with  a  DT&E  office 
recommendation  to  conduct  full  system  testing  prior  to  procuring 
additional  craft  during  initial  production.  The  DT&E  office  believed  the 
program  was  high  risk  because  the  Ship  to  Shore  Connector  was  a 
complete  redesign  of  a  previous  system  with  no  reuse  of  any  major 
component  (engines,  gearboxes,  hydraulics,  command  and  control 
software).  Navy  officials,  however,  believe  the  program  is  low  risk  since  it 
is  an  evolutionary  program  and  has  one  critical  technology,  a  fire 
suppression  system,  which  has  already  been  sufficiently  demonstrated 
and  qualified  through  test  and  evaluation.  In  addition,  the  Navy  estimated 
that  it  would  cost  $15  million  to  revise  the  existing  production  schedule  to 
accommodate  the  full  system  testing  as  recommended  by  the  DT&E 
office.  The  DT&E  office  and  the  Navy  reached  a  compromise  whereby 
OSD  would  review  available  system  test  results  before  more  craft  are 
authorized. 


Conclusions 


Current  fiscal  pressures,  along  with  the  threat  of  more  to  come,  have 
DOD  officials  looking  for  ways  to  increase  buying  power  by  controlling 
cost  and  schedule  overruns  on  weapon  acquisition  programs.  The  offices 
established  as  a  result  of  the  Reform  Act,  as  well  as  policy  provisions 
have  helped  DOD  make  inroads  towards  putting  weapon  acquisition 
programs  on  more  solid  footing.  Together,  the  offices  and  policy 
provisions  place  more  attention  on  requirements,  costs,  testing,  and 
reliability  as  early  as  Milestone  A.  The  provisions  of  the  act,  when 
specifically  focused  on  newer  programs,  are  having  a  positive  impact  on 
the  programs  and  the  acquisition  process.  They  show  that  expert 
attention  to  the  cost  and  achievability  of  capability  requirements,  the 
assumptions  made  for  cost  and  funding  of  programs,  and  the  amount  of 
systems  engineering  knowledge  that  is  brought  to  bear  early  make 
programs  more  executable. 
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Although  senior  officials  we  spoke  with  throughout  the  department  are 
receptive  to  the  broad  principles  of  the  Reform  Act,  it  is  too  early  to  tell  if 
the  Reform  Act  is  going  to  result  in  systemic  change  to  DOD’s  weapon 
acquisition  process.  DOD  faces  several  challenges  that  must  be 
addressed  to  get  lasting  change — organizational  capability  constraints, 
the  need  for  additional  cost  estimating  and  implementation  guidance,  the 
possibility  of  insufficient  systems  engineering  and  developmental  testing 
resources,  limited  dissemination  of  lessons  learned,  and  cultural  barriers 
between  OSD  and  the  services.  Some  challenges  appear  to  be  straight 
forward  to  address,  such  as  providing  guidance  for  estimating  operating 
and  support  costs,  providing  additional  guidance  for  conducting 
preliminary  design  reviews  and  competitive  prototyping  activities,  and 
disseminating  lessons  learned  to  the  broader  acquisition  community. 
However,  they  may  require  more  resources,  which  have  been  difficult  to 
obtain. 

For  Reform  Act  policies  and  practices  to  have  a  systemic  effect  across 
the  entire  portfolio  of  weapon  system  acquisition  programs,  the 
department  must  also  address  challenges  related  to  systems  engineering 
and  developmental  testing  resources  and  cultural  barriers  between  OSD 
and  the  services.  This  begins  with  the  services  identifying  key  leaders  at 
the  headquarters  level  and  within  program  offices  to  guide  systems 
engineering  and  developmental  testing  efforts  and  then  ensuring  that 
there  are  enough  trained  staff  to  carry  out  these  activities.  OSD  will  need 
to  continue  monitoring  the  services’  efforts.  It  will  also  require  an 
environment  where  the  services  stop  proposing  new  weapon  systems 
with  inflexible  requirements,  immature  technologies,  and  cost,  schedule, 
and  funding  assumptions  that  are  too  optimistic  at  the  start  of  a  program. 
Breaking  down  cultural  resistance  to  change  will  take  more  cooperation 
between  the  Under  Secretary  for  Acquisition,  Technology  and  Logistics 
and  other  OSD  offices,  and  service  acquisition  executives  to  address,  as 
well  as  continuity  of  leadership.  Efforts  by  the  PARCA  office  to  identify 
factors  that  correlate  to  good  or  poor  acquisition  outcomes,  particularly  as 
it  relates  to  program  manager  tenure,  will  be  beneficial.  The  services’ 
ability  to  demonstrate  that  the  Reform  Act  is  influencing  all  weapon 
acquisition  programs,  not  just  the  biggest,  will  be  a  key  indicator  for 
determining  whether  the  Reform  Act  has  had  a  positive  effect  on  DOD’s 
culture. 
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Recommendations  for  recommenc'  ^at  Secretary  of  Defense  take  the  following  four 

actions  to  enable  systemic  change  across  the  entire  portfolio  of  weapon 
Executive  Action  acquisition  programs: 

•  direct  the  Director  of  Cost  Assessment  and  Program  Evaluation  to 
issue  guidance  for  estimating  weapon  acquisition  program  costs  at 
Milestone  A  and  operating  and  support  costs  throughout  the 
acquisition  life  cycle  by  the  end  of  fiscal  year  2013  and  ensure  that  the 
office  prioritizes  its  resources  accordingly  to  accomplish  this  task; 

•  designate  responsibility  for  providing  advice  and  guidance  to  program 
offices  on  competitive  prototyping  and  preliminary  design  reviews  to 
the  appropriate  organization  within  the  Office  of  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology  and  Logistics  and  ensure  that 
the  guidance  is  developed.  The  office(s)  designated  would  be  the 
focal  point  for  addressing  program  office  issues  related  to  the  practical 
implementation  of  these  Reform  Act  provisions,  such  as  the  type  of 
competitive  prototyping  to  use,  the  timing  and  benefits  of  holding 
preliminary  design  reviews  prior  to  milestone  B,  and  if  a  preliminary 
design  review  should  be  held  after  milestone  B; 

•  direct  the  Deputy  Assistant  Secretaries  of  Defense  for  Systems 
Engineering  and  Developmental  Test  and  Evaluation  to  assess  and 
include  in  their  annual  report  to  the  Congress  beginning  with  the 
report  on  fiscal  year  2012  activities: 

•  the  extent  to  which  the  office  can  perform  their  required  activities 
with  allocated  resources; 

•  the  impact  budget  cuts  are  having  on  the  military  services  total 
workforce  (civilians,  military,  and  contractors)  and  ability  to  meet 
program  office  needs;  and 

•  progress  the  services  have  made  filling  leadership  positions,  such 
as  chief  engineers  at  the  service  level  and  technical  leads  for 
systems  engineering  and  developmental  testing  at  the  program 
office  level; 

•  direct  the  Director  of  Performance  Assessments  and  Root  Cause 
Analyses  to  make  lessons  learned  collected  during  its  root  cause 
analysis  evaluations  available  to  the  acquisition  workforce  and  ensure 
that  the  office  prioritizes  its  resources  accordingly. 


Agency  Comments 
and  Our  Evaluation 


DOD  provided  us  written  comments  on  a  draft  of  this  report.  DOD 
concurred  with  two  recommendations  and  partially  concurred  with  two 
others.  DOD’s  comments  appear  in  appendix  III.  DOD  also  provided 
technical  comments,  which  we  incorporated  as  appropriate  in  the  report. 
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DOD  agreed  with  the  intent  of  our  first  recommendation,  but  noted  that 
due  to  resource  constraints,  the  Cost  Assessment  and  Program 
Evaluation  office  could  not  guarantee  that  it  would  be  able  to  issue 
guidance  for  estimating  major  defense  acquisition  program  costs  at 
Milestone  A  and  operating  and  support  costs  throughout  the  acquisition 
lifecycle  by  the  end  of  fiscal  year  201 3.  We  continue  to  believe  that  the 
Cost  Assessment  and  Program  Evaluation  office  should  issue  the 
guidance  by  the  end  of  fiscal  year  201 3.  However,  if  that  is  not  possible 
from  a  resource  standpoint,  the  office  should  commit  to  a  date  and  devote 
the  resources  to  meeting  that  date.  We  will  continue  to  monitor  DOD’s 
efforts  to  develop  the  guidance. 

Although  DOD  concurred  with  our  second  recommendation,  we  revised 
this  recommendation  based  upon  discussions  with  DOD  officials  during 
the  agency  comment  period.  Our  revision  clarified  the  intent  of  this 
recommendation,  which  is  to  have  the  Secretary  of  Defense  designate  a 
specific  organization  within  the  Office  of  the  Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  to  provide  advice  and  guidance 
on  competitive  prototyping  and  preliminary  design  reviews.  We 
understand  that  the  department  has  issued  Reform  Act  implementation 
guidance  and  has  incorporated  aspects  of  competitive  prototyping  and 
preliminary  design  reviews  in  the  Defense  Acquisition  Guidebook. 

Further,  we  recognize  that  program  offices  we  visited  are  taking  steps  to 
implement  the  guidance  that  has  already  been  issued.  However,  based 
on  our  discussions  with  senior  level  officials,  we  believe  one  or  more 
offices  need  to  be  designated  with  the  responsibility  of  developing 
additional  guidance  and  answering  program  specific  questions  related  to 
the  practical  implementation  of  the  requirements.  As  noted  earlier  in  our 
report,  some  officials  questioned  when  to  use  prototyping  or  what  type  of 
prototyping  should  be  used.  In  addition,  there  were  questions  about  the 
timing  of  the  preliminary  design  reviews. 

DOD  partially  concurred  with  our  third  recommendation.  DOD  noted  that 
the  type  of  information  we  recommended  be  assessed  and  reported  on 
should  be  included  as  part  of  DOD’s  human  capital  strategic  planning 
process  and  as  such,  be  reported  in  DOD’s  annual  Acquisition  Workforce 
Strategic  Plan.  We  agree  that  the  impact  of  budget  cuts  on  the  workforce 
and  the  status  of  leadership  positions  could  be  addressed  in  the  annual 
strategic  plan.  However,  we  continue  to  believe  that  the  Deputy  Assistant 
Secretaries  for  Systems  Engineering  and  Developmental  Test  and 
Evaluation  should  include  an  assessment  in  their  joint  annual  report  to  the 
Congress  on  the  respective  offices’  ability  to  perform  activities  specified  in 
the  Reform  Act  with  available  resources. 
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DOD  concurred  with  our  fourth  recommendation,  which  would  make 
lessons  learned  from  root  cause  analyses  available  to  the  acquisition 
workforce. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense  and 
appropriate  Congressional  Committees.  In  addition,  the  report  is  available 
at  no  charge  on  the  GAO  website  at  http://www.gao.gov. 

If  you  have  any  questions  about  this  report  or  need  additional  information, 
please  contact  me  at  (202)  512-4841  or  sullivanm@gao.gov.  Contact 
points  for  our  Office  of  Congressional  Relations  and  Public  Affairs  may  be 
found  on  the  last  page  of  this  report.  Key  contributors  to  this  report  were 
Cheryl  Andrew,  Assistant  Director;  Laura  Greifner,  Julie  Hadley,  Megan 
Porter,  Rae  Ann  Sapp,  and  Marie  Ahearn. 


Michael  J.  Sullivan 
Director 

Acquisition  and  Sourcing  Management 
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Appendix  I:  Objectives,  Scope,  and 
Methodology 


This  report  examines  DOD’s  continued  implementation  of  the  Weapon 
Systems  Acquisition  Reform  Act  of  2009  (Reform  Act).  Specifically,  we 
examined  (1)  DOD’s  progress  in  implementing  Reform  Act  provisions; 

(2)  the  impact  the  Reform  Act  has  had  on  specific  acquisition  programs; 
and  (3)  challenges  remaining  in  improving  the  weapons  acquisition 
process. 

To  assess  DOD’s  progress  in  implementing  Reform  Act  provisions,  we 
interviewed  officials  and  analyzed  documents,  such  as  reports  to  the 
Congress  and  guidance  issued  from  the  Office  of  the  Secretary  of 
Defense  (OSD)  offices  of  the  (1)  Deputy  Assistant  Secretary  of  Defense 
for  Systems  Engineering  (SE),  (2)  Deputy  Assistant  Secretary  of  Defense 
for  Developmental  Test  and  Evaluation  (DT&E),  (3)  Cost  Assessment  and 
Program  Evaluation  (CAPE),  and  (4)  Performance  Assessments  and 
Root  Cause  Analyses  (PARCA)  to  determine  the  extent  to  which 
provisions  have  been  implemented.  We  focused  our  review  on  the  offices’ 
implementation  of  four  fundamental  Reform  Act  provisions:  developing 
policy  and  guidance;  approving  acquisition  documents;  monitoring 
programs  and  conducting  program  assessments;  and  developing 
performance  measures.  In  cases  where  provisions  had  not  been 
implemented,  we  asked  officials  about  the  reasons  for  the  delay  and  the 
expected  time  frame  for  completion.  We  also  interviewed  officials  and 
analyzed  documents  from  the  office  of  the  Assistant  Secretary  of  Defense 
for  Research  and  Engineering  and  the  Defense  Procurement  and 
Acquisition  Policy  office,  as  well  as  four  weapon  acquisition  programs, 
which  had  not  yet  started  development  to  determine  the  progress  DOD 
has  made  implementing  Reform  Act  provisions  related  to  preliminary 
design  reviews,  competitive  prototyping,  and  competition.  We  believe 
these  programs  offer  the  best  glimpse  at  how  the  OSD  offices  and 
Reform  Act  policies  are  influencing  acquisition  strategies.  The  weapon 
acquisition  programs  we  chose  for  this  analysis  were  part  of  a  larger  case 
study  review  that  is  described  below. 

To  determine  the  impact  the  Reform  Act  has  had  on  specific  weapon 
acquisition  programs,  we  selected  11  weapon  system  programs  to  use  as 
case  studies.  For  each  program  we  reviewed  relevant  program 
documentation  such  as  the  test  and  evaluation  master  plans, 
assessments  of  operational  test  readiness,  systems  engineering  plans, 
program  support  reviews,  root  cause  analyses,  analysis  of  alternatives 
reports  and  cost  estimates  as  applicable.  We  also  interviewed  appropriate 
program  officials  and  officials  from  the  OSD  offices  for  SE,  DT&E,  and 
CAPE  to  obtain  their  perspectives  about  (1)  the  level  of  interaction 
between  the  programs  and  OSD  offices;  (2)  changes  made  to  program 
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acquisition  strategies  as  a  result  of  interactions  with  the  OSD  offices;  and 
(3)  benefits  and  challenges  with  implementing  Reform  Act  provisions  on 
each  of  the  programs.  We  also  reviewed  the  performance  assessments 
and  root  cause  analyses  office’s  root  cause  analysis  documentation  for 
programs  that  incurred  Nunn-McCurdy  cost  or  schedule  breaches. 

We  selected  our  case  studies  based  on  input  from  the  officials  in  the  OSD 
offices  for  SE,  DT&E,  CAPE,  PARCA,  and  operational  test  and 
evaluation.  We  also  discussed  possible  case  studies  with  GAO 
employees  who  monitor  and  report  on  weapon  acquisition  programs  on 
an  annual  basis.  The  programs  we  selected  for  review  represent  a  variety 
of  platforms,  including  sea  vessels,  manned  and  unmanned  aircraft,  and 
land  systems.  Specifically,  we  examined  11  programs  at  various  stages  of 
the  acquisition  process.  Four  programs  had  not  yet  passed  Milestone  B, 
development  start,  at  the  time  we  began  our  review.  The  remaining  seven 
programs  had  completed  their  Milestone  B  review  and  were  in 
development  at  the  time  of  our  case  study  selection.  Of  the  seven 
programs,  three  have  breached  Nunn-McCurdy  cost  thresholds  since  the 
act  was  passed  and  have  had  to  satisfy  the  act’s  new  requirements  with 
regards  to  certification.  The  other  programs  had  significant  interaction 
with  one  or  more  of  the  OSD  offices  established  by  the  Reform  Act.  A 
complete  list  of  programs  is  provided  below. 


Table  4:  List  of  Programs  Selected  for  Case  Study  Review 

Programs  before  Milestone  B 

Service 

Platform 

Ground  Combat  Vehicle 

Army 

Ground 

Joint  Light  Tactical  Vehicle3 

Army/  Marines 

Ground 

Ohio  Class  Replacement 

Navy 

Sea 

Ship  to  Shore  Connector3 

Navy 

Sea 

Programs  after  Milestone  B 

Global  Hawk 

Air  Force 

Air 

Gray  Eagle 

Army 

Air 

Joint  Strike  Fighter 

Joint 

Air 

KC-46  Tanker 

Air  Force 

Air 

Littoral  Combat  Ship  Seaframe 

Navy 

Sea 

Remote  Minehunting  System 

Navy 

Sea 

Small  Diameter  Bomb  Increment  II 

Air  Force 

Air 

Source:  GAO  analysis. 

aDuring  the  course  of  our  review,  the  Joint  Light  Tactical  Vehicle  and  Ship  to  Shore  Connector 
programs  held  a  Milestone  B. 
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While  our  sample  of  1 1  case  studies  allowed  us  to  learn  about  the  impact 
the  Reform  Act  offices  have  had  on  DOD  acquisitions,  it  was  designed  to 
provide  anecdotal  information,  not  findings  that  would  be  representative 
of  all  the  department’s  weapon  acquisition  programs. 

To  determine  challenges  remaining  in  improving  defense  acquisitions  we 
relied  on  information  we  collected  and  analyzed  during  our  case  study 
review  of  11  weapon  acquisition  programs.  We  also  solicited  the  opinions 
of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics;  other  senior  level  officials  in  the  Office  of  the  Secretary  of 
Defense  including  the  leaders  of  each  of  the  offices  created  as  a  result  of 
the  Reform  Act;  as  well  as  the  military  services’  Senior  Acquisition 
Executives. 

We  conducted  this  performance  audit  from  January  2012  to  December 
2012  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 
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in  Implementing  Weapon  System  Acquisition 
Reform  Act  Provisions 

Table  5:  Implementation  of  Select  Reform  Act  Provisions-  Systems  Engineering 

Reform  Act  Provisions 

Status 

Develop  policy  and  guidance 

Complete:  Issued  a  reliability,  availability,  and  maintainability  Directive-Type 
Memorandum,  a  development  planning  Directive-Type  Memorandum,  and  a  DOD 
Instruction  for  DASD  (Systems  Engineering),  participated  in  JCIDS  revisions,  developed 
guidance  for  incorporating  systems  engineering  into  development  contracts,  streamlined 
the  Systems  Engineering  Plan  and  Program  Protection  Plan,  and  released  an  update  to 
the  Defense  Acquisition  Guidebook,  chapter  4. a  Continue  to  refine  policies  and 
guidance  as  necessary. 

Approve  documents 

Completing  on  annual  basis:  Approved  52  Systems  Engineering  Plans  since  2009, 
including  1 5  in  fiscal  year  2011. 

Monitor  programs  /  conduct  assessments 

Completing  on  annual  basis:  Review  portfolio  of  234  programs.  In  fiscal  year  201 1 , 
participated  in  73  overarching  integrated  product  team  meetings,  6  peer  reviews  of 
acquisition  contracts,  and  conducted  15  Program  Support  Reviews. 

Develop  performance  measures 

Complete:  Developed  a  set  of  time-based  metrics  to  assess  each  program’s  ability  to 
execute  its  system  engineering  plans  and  address  risks  the  office  had  identified  in  prior 
reviews.  The  metrics  measure  program  cost,  schedule,  staffing,  reliability,  availability 
and  maintainability,  software,  integration,  performance  and  manufacturing,  and  are  to  be 
incorporated  into  each  program’s  systems  engineering  plan  and  evaluated  at  various 
points  in  the  development  process. 

Source:  GAO  analysis  of  DOD  data. 

a  DOD  Directive-Type  Memorandum  (DTM)  1 1-003, ’’Reliability  Analysis,  Planning,  Tracking,  and  Reporting,”  (Mar.  21,  2011);  DOD 

DTM  10-017,  “Development  Planning  to  Inform  Materiel  Development  Decision  Reviews  and  Support  of  Analysis  of  Alternatives,” 

(Sept.  13,  2010,  Incorporating  Change  2,  Dec.  9,  2011);  DOD  Instruction  5134.16,  “Deputy  Assistant  Secretary  of  Defense  for  Systems 
Engineering  (DASD(SE)),”  (Aug.  19,  2011). 

Table  6:  Implementation  of  Select  Reform  Act  Provisions-  Developmental  Test  and  Evaluation 

Reform  Act  Provisions 

Status 

Develop  policy  and  guidance 

Complete:  Updated  guide  for  incorporating  test  and  evaluation  into  acquisition  contracts. 
Championed  updates  to  DOD  Instruction  assigning  responsibilities  and  authorities  to 
Developmental  Test  and  Evaluation  office,  which  is  in  the  process  of  being  updated. 
Updated  guidance  to  include  reliability  factors  in  the  Test  and  Evaluation  Master  Plan. 
Continue  to  refine  policies  and  guidance  as  necessary. 

Approve  documents 

Completing  on  annual  basis:  Reviewed  and  approved  186  Test  and  Evaluation  Master 
Plans  since  2009,  including  44  in  fiscal  year  201 1 . 

Monitor  programs  /  conduct  assessments 

Completing  on  annual  basis:  Review  portfolio  of  nearly  250  programs.  In  fiscal  year  201 1 , 
participated  in  22  defense  acquisition  board  meetings  and  59  overarching  integrated 
product  team  meetings.  The  office  has  also  conducted  16  Assessment  of  Operational 

Test  Readiness  reviews  since  2009. 

Develop  performance  measures 

In  process:  Piloted  performance  measures  on  two  programs.  The  measures  were  then 
updated  and  are  being  applied  to  over  40  programs  that  were  selected  for  reporting  in  the 
fiscal  year  2012  joint  annual  report.  The  assessments  are  being  used  to  support  the  write¬ 
up  of  the  program  engagement  section. 

Source:  GAO  analysis  of  DOD  data. 
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Reform  Act  Provisions 

Table  7:  Implementation  of  Select  Reform  Act  Provisions-  Cost  Assessment  and  Program  Evaluation 

Reform  Act  Provisions 

Status 

Develop  policy  and  guidance 

In  process:  Issued  first  policy  document  in  May  2012,  which  is  the  basis  for  additional 
policy  documents.  Updating  its  Operating  and  Support  Cost  Estimating  Guidebook,  which 
will  address  the  Reform  Act  requirement  for  DOD  to  issue  guidance  related  to  full 
consideration  of  life  cycle  management  and  sustainability  costs  in  major  defense 
acquisition  programs. 

Approve  documents 

Not  applicable:  The  Reform  Act  does  not  require  that  the  office  approve  acquisition 
documents. 

Monitor  programs  /  conduct  assessments 

Completing  on  annual  basis:  Conducted  independent  cost  assessments  for  Milestone  A 
and  B  certification  on  30  future  and  current  major  defense  acquisition  programs,  since 
2009,  including  8  in  fiscal  year  201 1 .  The  office  conducted  3  Milestone  C  and  3  Nunn- 
McCurdy  certification  reviews  in  fiscal  year  201 1 . 

Develop  performance  measures 

Not  applicable:  The  Reform  Act  does  not  require  that  the  office  develop  performance 
measures. 

Source:  GAO  analysis  of  DOD  data. 

Table  8:  Implementation  of  Select  Reform  Act  Provisions-  Performance  Assessments  and  Root  Cause  Analyses 

Reform  Act  Provisions 

Status 

Develop  policy  and  guidance 

In  process:  Developing  guidance  to  assist  offices  in  conducting  root  cause  analyses.  The 
guidance  for  conducting  performance  assessments  is  expected  to  be  released  in  early 
fiscal  year  201 3. 

Approve  documents 

Not  applicable:  The  Reform  Act  does  not  require  that  the  office  approve  acquisition 
documents. 

Monitor  programs  /  conduct  assessments 

Completing  on  annual  basis:  Completed  14  Root  Cause  Analyses  for  programs  which 
have  undergone  a  Nunn-McCurdy  breach  or  were  requested  by  OSD  and  has  completed 

26  semi-annual  follow-up  reports  on  these  programs.  Providing  OSD  with  the  execution 
status  of  DOD’s  portfolio  of  acquisition  programs  through  the  Defense  Acquisition 

Executive  Summary  process. 

Develop  performance  measures 

In  process:  Utilize  Defense  Acquisition  Executive  Summary  information  to  identify  cost 
performance,  schedule,  funding,  and  technical  performance  issues  on  Major  Defense 
Acquisition  Programs.  Continuing  to  develop  performance  measures. 

Source:  GAO  analysis  of  DOD  data. 
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ACQUISITION. 
TECHNOLOGY 
AND  LOGISTICS 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON.  DC  20301-3000 


DEC  12 


Mr.  Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Sullivan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  Government  Accountability 
Office  (GAO)  Draff  Report,  GAO-13-103,  “WEAPONS  ACQUISITION  REFORM:  Reform  Act 
is  Helping  DoD  Acquisition  Programs  Reduce  Risk,  but  Implementation  Challenges  Remain,” 
dated  November  5, 2012  (GAO  Code  121024).  Detailed  comments  on  the  report 
recommendations  are  enclosed. 


Sincerely, 


Nancy  L.  Spruill 
Director 

Acquisition  Resources  &  Analysis 


Enclosures: 
As  stated 
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GAO  DRAFT  REPORT  DATED  NOVEMBER  5,  2012 
GAO-13-103  (GAO  CODE  121024) 

“WEAPONS  ACQUISITION  REFORM:  REFORM  ACT  IS  HELPING 
DOD  ACQUISITION  PROGRAMS  REDUCE  RISK,  BUT 
IMPLEMENTATION  C  HALLENGES  REMAIN" 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary'  of  Defense 
to  direct  the  Director  of  Cost  Assessment  and  Program  Evaluation  to  issue 
guidance  for  estimating  weapon  acquisition  program  costs  at  Milestone  A  and 
operating  and  support  costs  throughout  the  acquisition  lifecycle  by  the  end  of 
fiscal  year  2013  and  ensure  that  that  the  office  prioritizes  its  resources 
accordingly.  (See  pages  18  through  19/GAO  Draft  Report.) 

DoD  RESPONSE:  Partially  concur. 

The  office  of  Cost  Assessment  and  Program  Evaluation  (CAPE)  agrees 
wholeheartedly  with  the  intent  of  GAO's  recommendation  and  is  in  the  process  of 
updating  its  guidance  for  cost  estimation,  including  guidance  for  estimating  major 
defense  acquisition  program  costs  at  Milestone  A  and  operating  and  support  costs 
throughout  the  acquisition  lifecycle.  However,  due  to  resource  constraints,  CAPE 
cannot  guarantee  that  the  guidance  will  be  issued  in  accordance  with  the  schedule 
recommended  by  GAO. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense 
to  designate  responsibility  for  providing  advice  and  guidance  on  competitive 
prototyping  and  preliminary  design  reviews  to  the  appropriate  Office  of  the 
Secretary  of  Defense  (OSD)  organization  and  ensure  that  that  guidance  is 
developed.  (Seepage  19/GAO  Draft  Report.) 

DoD  RESPONSE:  Concur. 

The  WSARA  Implementation  DTM  09-027  assigned  responsibilities  for  the 
Preliminary  Design  Review  (PDR)  and  Competitive  Prototyping  to  the  Under 
Secretary  of  Defense  (Acquisition.  Technology,  and  Logistics)  (USD(AT&L)). 

•  PDR  the  DTM  requires  a  PDR,  including  a  post-PDR  report  to  the 
USD(AT&L),  at  Milestone  B.  Additionally,  in  accordance  with  the  April 
20,  201 1  USD(AT&L)  Memorandum  "Document  Streamlining  Program 
Strategies  and  Systems  Engineering  Plan,"  a  program's  PDR  is  to  be 
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documented  in  a  program's  systems  engineering  plan  (SEP)  that  will  be 
approved  by  the  Deputy  Assistant  Secretary  of  Defense  (Systems 
Engineering).  The  SEP  outline  and  guidance  for  the  PDR  can  be  found  in 
Chapter  4  of  the  Defense  Acquisition  Guidebook. 

•  Competitive  Prototyping  -  in  addition  to  being  required  by  the  DTM,  the 
competitive  prototyping  approach  for  an  individual  program  is  required  to 
be  documented  in  its  Technology  Development  Strategy  (TDS)/Acquisition 
Strategy  (AS)  which  will  be  approved  by  the  USD(AT&L).  The  TDS/AS 
Outline  contains  guidance  for  competitive  strategies  (including  competitive 
prototyping  plans)  and  can  be  found  in  the  Defense  Acquisition  Guidebook. 

Responsibilities  and  guidance  for  PDRs  and  competitive  prototyping  will  also  be 
reflected  in  the  upcoming  revision  of  DoDI  5000.02.  Additionally,  competitive 
prototyping  is  included  as  an  action  item  in  the  department's  Better  Buying  Power 
2.0  initiatives  and  will  continue  to  be  addressed  by  the  USD(AT&L). 

RECOMMENDATION  3:  The  GAO  recommends  that  the  Secretary  of  Defense 
direct  the  Deputy  Assistant  Secretaries  for  Systems  Engineering  and 
Developmental  Test  and  Evaluation  to  assess  and  include  in  their  annual  report  to 
the  Congress  beginning  with  the  report  on  fiscal  year  2012  activities: 

a.  the  extent  to  which  the  office  can  perform  their  required  activities  with 
allocated  resources; 

b.  the  impact  budget  cuts  are  having  on  the  military  services  total  workforce 
(civilians,  military,  and  contractors)  and  ability  to  meet  program  office 
needs;  and 

c.  the  progress  the  services  have  made  filling  leadership  positions,  such  as 
chief  engineers  at  the  headquarters  level  and  technical  leads  for  systems 
engineering  and  developmental  testing  at  the  program  office  level. 

(See  page  19/GAO  Draft  Report.) 

DoD  RESPONSE:  Partially  concur. 

The  information  requested  is  linked  to  human  capital  strategic  planning  in  the 
systems  engineering  and  testing  functional  career  areas.  Recommend  that  the 
Deputy  Assistant  Secretaries  for  Systems  Engineering  and  Developmental  l  est 
and  Evaluation  work  with  the  Director,  Human  Capital  Initiatives  to  collect  and 
assess  the  data  and  include  as  part  of  the  annual  Acquisition  Workforce  Strategic 
Plan  rather  than  as  part  of  the  joint  Systems  Engineering  and  Developmental  Test 
and  Evaluation  annual  report. 
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RECOMMENDATION  4:  The  GAO  recommends  that  the  Secretary  of  Defense 
to  direct  the  Director  of  Performance  Assessment  and  Root  Cause  Analyses  to 
make  lessons  learned  collected  during  its  root  cause  analysis  evaluations  available 
to  the  acquisition  workforce  and  ensure  that  the  office  prioritizes  its  resources 
accordingly.  (Seepage  19/GAO  Draft  Report.) 

DoD  RESPONSE:  Concur. 


(121024) 
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The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 
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GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (http://www.gao.gov).  Each  weekday 
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